And just like that, it was over…
News recently came out that Pokémon Go – the augmented reality app that had dominated global consciousness and phone screens since early July – had fallen from its perch atop Apple’s App Store. This, coupled with the prior week’s news that the game’s paying user base had dropped 79% since its peak, confirmed the game’s rapid decline.
So what does this mean for small- and medium-sized businesses, many of which saw revenues skyrocket as a result of players loitering outside of their businesses? Contrary to most reports out there, not as much as you might think.
There’s no questioning the game’s impact
Pokemon Go was a global phenomenon that left an indelible impact. Below are just a few of the accolades the game boasted:
- Most revenue grossed by a mobile game in its first month ($206.5 million in first month)
- Most downloaded mobile game in its first month (130 million downloads)
- Most countries in which mobile game was simultaneously ranked the most downloaded app (was #1 in 70 countries)
- Most countries in which mobile game was simultaneously ranked the highest grossing app (was #1 in 55 countries)
- Fastest mobile game to gross $100 million (a mere 20 days)
What may be even more impressive than these records was the way the game got people outside to experience actual sunlight (a concept that has become more and more foreign to each passing generation). With all of these players wandering the streets, news outlets began reporting on businesses seeing staggering increases in revenue.
Unfortunately there’s a difference between a few one-off observations and true causal impact. Were some businesses impacted? Absolutely. But merchants as a whole did not see revenues affected by the game.
Our super geeky analysis
(WARNING: readers who are allergic to math may wish to skip this section)
In the hopes of complementing the feel-good stories with hard statistics, our crack team of Data Scientists was locked in a room with a bottle of water, a bucket, and a significant sample of revenues from the company’s database of over 2 million merchants. We were promised great riches (aka new calculators and pocket protectors) if we could just put a number to Pokémon’s economic impact. Unfortunately, even this motivation was not enough for us to uncover anything meaningful.
Before we continue, it is important that we review the concept of hypothesis testing for the uninitiated. For those of you familiar with America’s judicial system, the notion of “innocent until proven guilty” should ring a bell. In statistics, there is a similar concept: “same until proven ‘probably not the same.’” This may sound like semantics to you – it may even sound stupid – but this is how statisticians think.
This means we start with the assumption that the revenue gains of merchants near PokéStops are the same as revenue gains of merchants not near PokéStops – holding all else constant – unless proven otherwise.
Another very important concept to know is random sampling. Samples are assumed to be somewhat representative of an entire group (dubbed the “population” in statistical parlance). This means that if you knowingly subset your data into a specific (i.e. non-random) group, you essentially negate any study’s result from being representative of the rest of the population.
So to recap, in order to statistically determine that PokéStops had an impact on merchants’ revenues, our task was to disprove that their revenue gains were the same as merchants not near PokéStops.
Our methods
WARNING: readers who are allergic to math will REALLY regret reading this section
Ceteris paribus
In Latin, ceteris paribus means “all other things being equal.” Our first task was to normalize the data so that all measurements of merchants’ performance could be compared “apples to apples.” Our analysis included adjustments for:
- Annual seasonality trends
- Day of week impact
- Merchant size
- Segment growth rates
Comparison of means
The most naïve way of comparing the impact is to measure the mean (aka “average”) percent change in revenue. To compute this, we evaluated merchants’ revenues during the six weeks prior to Pokémon’s launch and compared them to revenues after Pokémon’s launch. Using what is called a “Student’s t-test,” we were unable to note any difference between merchants near PokéStops and those not near one.
Comparison of distributions
A second way of analyzing the PokéStop merchants versus non-PokéStop merchants is to compare the distributions of their revenue changes. Just because the averages of their percent-revenue-change were the same does not guarantee that the samples still didn’t come from different populations. In the example below, we’ve depicted what it might look like if we were to sample from two different groups that happened to have the same average value:
In the example shown, the groups have different standard deviations, but they could theoretically exhibit any number of differences including skewedness, kurtosis, fat-tails, and several other funny sounding names.
Identifying differences in the distributions (such as the ones just listed) would be another indication that PokéStop and non-PokéStop merchants were indeed different. To do this we combined two methods.
First, we estimated the distribution of PokéStop and non-PokéStop merchant percent-revenue-changes using a method called Kernel Density Estimates (KDE). Below is the output of our sample distributions and density estimates:
You might be noting to yourself that the two distribution estimates aren’t exactly the same. Luckily, there is a statistical test to determine whether they are different enough to be considered meaningfully different. The Kolmogorov-Smirnov Test is a non-parametric test of the equality for one-dimensional probability distributions. Using this test, we can say that it is nearly impossible that these two samples come from different populations. To put another way, we still have no evidence that the PokéStop merchants and non-PokéStop merchants are different from one another.
Diffusion regression state-space model
If you’re still reading this, your commitment to understanding our methods is commendable. As a reward, you get to learn about the coolest sounding of all of our analyses: diffusion regression state-space modeling. This method utilizes Dynamic Time-Warping (DTW) to determine whether a latent variable (aka an unknown external factor) exists that could differentiate two time series. In this case, our latent variable was the potential impact of a nearby PokéStop (NOTE: if that made sense to you, we highly encourage you to go to our “Jobs” page and apply to a Data Science position).
Without getting into the gory details, we were once again unable to differentiate between PokéStop and non-PokéStop merchants.
Conclusion
Look, we didn’t set out to disagree with the world; we seek acceptance just like everyone else. In fact, we performed the analysis nearly two months ago with the hopes of being the first to publish a report on the game’s economic impact. When we came up empty, we figured there was no point publishing our results.
However, it was brought to our attention that other companies were communicating conclusions that we unequivocally disagreed with, and we were forced into responding. Womply’s mission is to use technology and data to grow, protect, and simplify small business, and we believe that poorly performed analyses have established a (potentially expensive) false belief about the impact of future augmented reality games.
The Pokémon craze may be over, but other augmented reality games are sure to come. Game developers will almost assuredly exploit merchants who believe that spending money on lures will increase business.
If this post was not enough to explain why we think merchants should ignore the hype (or at least proceed with extreme caution), our next post should quash any doubt about what you’ve read in the news.