“God, grant me the serenity to accept the things I cannot change,
Courage to change the things I can,
And wisdom to know the difference.”
These are the famous words written by American theologian Reinhold Niebuh. Despite their association with various twelve-step recovery programs, the quote holds broad applicability, regardless of religious affiliation.
Small- and medium-sized business owners are tasked with countless responsibilities to grow their businesses. Yet owners don’t have teams of business strategists and data scientists making optimized recommendations. They have to rely on their own experience and intuition to do what they think is right. Often times, this is informed by what they see and read in the media.
Get smarter about your customers and their impact on your business, and total visibility into your revenue trends with Womply’s business intelligence software. Learn more, plus get free reputation monitoring and customer insights when you sign up for Womply Free!
The problem is that the media either 1) doesn’t understand the complexity of statistics to interpret data properly, or 2) simply doesn’t care about the interpretation as long as it generates eyeballs on their website.
Pokémon Go
Our last post reported about the truth behind Pokémon Go’s impact on businesses. Despite the seemingly unending stories, Pokémon Go did not create the economic boon most news outlets reported (for more details about how we drew this conclusion, check out our in-depth analysis).
This flies in the face of the feel good stories and analyses that have been written to date. In light of this, we took painstaking measures to analyze a significant sample from our database of over 2 million merchants to ensure we believed what we reported.
So what about the struggling ice cream shop in Washington state that saw its sales triple as a result of Pokémon players? And did the $10 in Pokémon lures not yield this New York pizza shop the 75% boost in sales it boasted?
And what about the infographic put together by Revel Systems that made spectacular sounding claims that included:
- Among merchants using Revel software, 63% who reported being near PokéStops saw their weekly sales increase
- Of that 63%, the average revenue increase seen was 12%
- The average increase in weekly gross sales totaled more than $2,000 per business
Well, let’s break things down.
Data Analysis 101
It is critical to start by recognizing the significant difference between the following two statements:
- Merchants near PokéStops saw an increase in revenues
- Being near a PokéStop resulted in an increase in revenues
If these sound the same to you, you now understand how analyses can easily be miscommunicated in the news. Writers who aren’t versed in the nuances of statistics can often misinterpret facts.
(Note: statement #1 above implies correlation, while statement #2 implies causation)
Let’s just get this out of the way. Can being near a PokéStop increase a merchant’s revenues? The answer to that is an unequivocal yes. Below is a revenue graph for an anonymous quick serve restaurant in Newport Beach, CA, which also happened to be a PokéStop.
It is undeniable that this restaurant saw significant revenue gains for the period of time when the game was popular. Given the comments on review sites, as well as social media, we know this to be the result of the game.
If you’ve concluded from the above graphic that Pokémon Go increases revenues, let me present to you the following analogy: think about the last weight loss infomercial that you saw (don’t worry, nobody here is judging).
Did the people shown lose weight? Absolutely. Was the weight loss attributable to the pills they were taking? It’s possible. Are you going to lose weight if you take those pills? It’s highly unlikely.
You’ve seen enough infomercials to know that the vast majority are scams. They cherry pick extreme results and present them to you as truth. This is the exact same thing news outlets have done with Pokémon stories. The confusion stems from the fact that these stories are coming from reputable publications and sites, not infomercials that we inherently distrust.
So what about the analysis done by Revel Systems? Surely their statistics can confirm the “Pokémon Effect.” The problem with their analysis is that it was biased and lacked context.
When 63% of merchants reported an increase in sales, what was this in comparison to? Their study only included merchants in Chicago, San Francisco, and New York City – all of which see peak tourist volume in July (the month Pokémon Go was released). Merchants in these cities would have seen strong revenues regardless of the game’s presence. Below is a graph contrasting Revel System’s figure with our estimate of non-PokeStops:
Amongst merchants in these cities that weren’t PokéStops we found that 61% saw revenue gains. Given the sampling error that exists, the difference between PokéStops and non-PokéStops is statistically insignificant. Furthermore, Revel Systems indicated that merchants that did see gains saw an average increase of 12%. We found that the average merchant amongst non-PokéStops saw an 11.2% gain – once again, a statistically insignificant difference.
Why are we making such a fuss?
You may be thinking to yourself, “these are supposed to be fun analyses related to an interesting topic… why is Womply ruining the fun?”
To be clear, if one-off stories and infographics was where it all stopped, we would probably be playing along in the fun too. Unfortunately, everyone has taken Revel’s superficial analysis and begun making recommendations about how to drive business. Revel System’s CTO dangerously implied a $2,000 increase in weekly business for any merchant willing to participate in this new virtual world.
Additionally, highly respected news outlets, such as Forbes, CNBC, and inc.com, have all published articles about how to make money through augmented reality. These words will undoubtedly influence business owners moving forward.
Pokémon Go was the first of many augmented reality games, and its success exceeded anybody’s wildest imagination. Rest assured that game makers have taken notice and will capitalize on expectations the next time around. While players will likely remain the key revenue contributor for future games, there is no doubt that game makers will cash in (aka significantly increase prices) on the perception that game spots drive merchant revenue – and based on current prevailing perceptions, merchant owners will gladly pay.
Our goal isn’t to pick on the flaws of someone else’s analysis. We have every reason to believe that their statistics are factually correct. Our issue rests with how the data was interpreted: without context and implying causality. Unfortunately, with the way news is communicated and perpetuated today, the thoughts of a few misinformed can perpetuate throughout the world like a giant game of telephone at the speed of social media.
Channeling our inner Reinhold Niebuh, we hope that if you take away three things from this post, it would be that 1) There are often external economic factors you cannot change, 2) There are investments that will actually make a difference, and 3) Real statistics can grant you the wisdom to know the difference.