Pokémon Go – Avoiding the Pitfalls of Media-Driven Statistics

“God, grant me the serenity to accept the things I cannot change,
Courage to change the things I can,
And wisdom to know the difference.”

These are the famous words written by American theologian Reinhold Niebuh. Despite their association with various twelve-step recovery programs, the quote holds broad applicability, regardless of religious affiliation.

Small- and medium-sized business owners are tasked with countless responsibilities to grow their businesses. Yet owners don’t have teams of business strategists and data scientists making optimized recommendations. They have to rely on their own experience and intuition to do what they think is right. Often times, this is informed by what they see and read in the media.

Get smarter about your customers and their impact on your business, and total visibility into your revenue trends with Womply’s business intelligence software. Learn more, plus get free reputation monitoring and customer insights when you sign up for Womply Free!

The problem is that the media either 1) doesn’t understand the complexity of statistics to interpret data properly, or 2) simply doesn’t care about the interpretation as long as it generates eyeballs on their website.

Pokémon Go

Our last post reported about the truth behind Pokémon Go’s impact on businesses. Despite the seemingly unending stories, Pokémon Go did not create the economic boon most news outlets reported (for more details about how we drew this conclusion, check out our in-depth analysis).

This flies in the face of the feel good stories and analyses that have been written to date. In light of this, we took painstaking measures to analyze a significant sample from our database of over 2 million merchants to ensure we believed what we reported.

So what about the struggling ice cream shop in Washington state that saw its sales triple as a result of Pokémon players? And did the $10 in Pokémon lures not yield this New York pizza shop the 75% boost in sales it boasted?

And what about the infographic put together by Revel Systems that made spectacular sounding claims that included:

  • Among merchants using Revel software, 63% who reported being near PokéStops saw their weekly sales increase
  • Of that 63%, the average revenue increase seen was 12%
  • The average increase in weekly gross sales totaled more than $2,000 per business

Well, let’s break things down.

Data Analysis 101

It is critical to start by recognizing the significant difference between the following two statements:

  1. Merchants near PokéStops saw an increase in revenues
  2. Being near a PokéStop resulted in an increase in revenues

If these sound the same to you, you now understand how analyses can easily be miscommunicated in the news. Writers who aren’t versed in the nuances of statistics can often misinterpret facts.

(Note: statement #1 above implies correlation, while statement #2 implies causation)

Let’s just get this out of the way. Can being near a PokéStop increase a merchant’s revenues? The answer to that is an unequivocal yes. Below is a revenue graph for an anonymous quick serve restaurant in Newport Beach, CA, which also happened to be a PokéStop.

It is undeniable that this restaurant saw significant revenue gains for the period of time when the game was popular. Given the comments on review sites, as well as social media, we know this to be the result of the game.

If you’ve concluded from the above graphic that Pokémon Go increases revenues, let me present to you the following analogy: think about the last weight loss infomercial that you saw (don’t worry, nobody here is judging).

Did the people shown lose weight? Absolutely. Was the weight loss attributable to the pills they were taking? It’s possible. Are you going to lose weight if you take those pills? It’s highly unlikely.

You’ve seen enough infomercials to know that the vast majority are scams. They cherry pick extreme results and present them to you as truth. This is the exact same thing news outlets have done with Pokémon stories. The confusion stems from the fact that these stories are coming from reputable publications and sites, not infomercials that we inherently distrust.

So what about the analysis done by Revel Systems? Surely their statistics can confirm the “Pokémon Effect.” The problem with their analysis is that it was biased and lacked context.

When 63% of merchants reported an increase in sales, what was this in comparison to? Their study only included merchants in Chicago, San Francisco, and New York City – all of which see peak tourist volume in July (the month Pokémon Go was released). Merchants in these cities would have seen strong revenues regardless of the game’s presence. Below is a graph contrasting Revel System’s figure with our estimate of non-PokeStops:

Amongst merchants in these cities that weren’t PokéStops we found that 61% saw revenue gains. Given the sampling error that exists, the difference between PokéStops and non-PokéStops is statistically insignificant. Furthermore, Revel Systems indicated that merchants that did see gains saw an average increase of 12%. We found that the average merchant amongst non-PokéStops saw an 11.2% gain – once again, a statistically insignificant difference.

Why are we making such a fuss?

You may be thinking to yourself, “these are supposed to be fun analyses related to an interesting topic… why is Womply ruining the fun?”

To be clear, if one-off stories and infographics was where it all stopped, we would probably be playing along in the fun too. Unfortunately, everyone has taken Revel’s superficial analysis and begun making recommendations about how to drive business. Revel System’s CTO dangerously implied a $2,000 increase in weekly business for any merchant willing to participate in this new virtual world.

Additionally, highly respected news outlets, such as ForbesCNBC, and inc.com, have all published articles about how to make money through augmented reality. These words will undoubtedly influence business owners moving forward.

Pokémon Go was the first of many augmented reality games, and its success exceeded anybody’s wildest imagination. Rest assured that game makers have taken notice and will capitalize on expectations the next time around. While players will likely remain the key revenue contributor for future games, there is no doubt that game makers will cash in (aka significantly increase prices) on the perception that game spots drive merchant revenue – and based on current prevailing perceptions, merchant owners will gladly pay.

Our goal isn’t to pick on the flaws of someone else’s analysis. We have every reason to believe that their statistics are factually correct. Our issue rests with how the data was interpreted: without context and implying causality. Unfortunately, with the way news is communicated and perpetuated today, the thoughts of a few misinformed can perpetuate throughout the world like a giant game of telephone at the speed of social media.

Channeling our inner Reinhold Niebuh, we hope that if you take away three things from this post, it would be that 1) There are often external economic factors you cannot change, 2) There are investments that will actually make a difference, and 3) Real statistics can grant you the wisdom to know the difference.

Recent Articles

9 Atlanta area events that can be great for your small business in 2022

In this 5-minute read: 9 events in Atlanta, GA in 2022 that your business should know about Start your year …

Read More

25 free business listing websites every small or local business should be on (Updated for 2022!)

Be sure to claim your company’s free listings on these sites so more customers can find your business. 

Read More

9 events that every Houston, TX area small business should know about in 2022

In this 5-minute read: 9 local events that Houston small businesses should take advantage of this year Attending events, business …

Read More

Small business events in Chicago: 8 ways to grow and market your business in 2022

In this 5-minute read: 8 Chicago events to add to your 2022 business calendar Wherever your business is located, you …

Read More

10 2022 Washington DC area events that can be great for your small business

In this 5-minute read: 10 events that Washington DC area businesses need to know about this year Which DC events …

Read More

5 Orlando area events in 2022 your small business should take advantage of

In this 5-minute read: 5 2022 events in Orlando, FL that your business needs to know about Which events may …

Read More

See why Womply is the #1 marketing and CRM solution used by 500,000+ businesses.