Pokémon Go - Avoiding the Pitfalls of Media-Driven Statistics

“God, grant me the serenity to accept the things I cannot change,
Courage to change the things I can,
And wisdom to know the difference.”

These are the famous words written by American theologian Reinhold Niebuh. Despite their association with various twelve-step recovery programs, the quote holds broad applicability, regardless of religious affiliation.

Small- and medium-sized business owners are tasked with countless responsibilities to grow their businesses. Yet owners don’t have teams of business strategists and data scientists making optimized recommendations. They have to rely on their own experience and intuition to do what they think is right. Often times, this is informed by what they see and read in the media. 

The problem is that the media either 1) doesn’t understand the complexity of statistics to interpret data properly, or 2) simply doesn’t care about the interpretation as long as it generates eyeballs on their website.

Pokémon Go

Our last post reported about the truth behind Pokémon Go’s impact on businesses. Despite the seemingly unending stories, Pokémon Go did not create the economic boon most news outlets reported (for more details about how we drew this conclusion, check out our in-depth analysis).

This flies in the face of the feel good stories and analyses that have been written to date. In light of this, we took painstaking measures to analyze a significant sample from our database of over 2 million merchants to ensure we believed what we reported.

So what about the struggling ice cream shop in Washington state that saw its sales triple as a result of Pokémon players? And did the $10 in Pokémon lures not yield this New York pizza shop the 75% boost in sales it boasted?

And what about the infographic put together by Revel Systems that made spectacular sounding claims that included:

  • Among merchants using Revel software, 63% who reported being near PokéStops saw their weekly sales increase
  • Of that 63%, the average revenue increase seen was 12%
  • The average increase in weekly gross sales totaled more than $2,000 per business

 Well, let’s break things down.

Data Analysis 101

It is critical to start by recognizing the significant difference between the following two statements:

  1. Merchants near PokéStops saw an increase in revenues
  2. Being near a PokéStop resulted in an increase in revenues

If these sound the same to you, you now understand how analyses can easily be miscommunicated in the news. Writers who aren’t versed in the nuances of statistics can often misinterpret facts.

(Note: statement #1 above implies correlation, while statement #2 implies causation)

Let’s just get this out of the way. Can being near a PokéStop increase a merchant’s revenues? The answer to that is an unequivocal yes. Below is a revenue graph for an anonymous quick serve restaurant in Newport Beach, CA, which also happened to be a PokéStop.

It is undeniable that this restaurant saw significant revenue gains for the period of time when the game was popular. Given the comments on review sites, as well as social media, we know this to be the result of the game.

If you’ve concluded from the above graphic that Pokémon Go increases revenues, let me present to you the following analogy: think about the last weight loss infomercial that you saw (don’t worry, nobody here is judging). This image below probably looks pretty familiar.

Credit: http://www.slideshare.net/viafortune/herbalife-presentation-22629522

Credit: http://www.slideshare.net/viafortune/herbalife-presentation-22629522

Did the person shown lose weight? Absolutely. Was the weight loss attributable to the pills he was taking? It’s possible. Are you going to lose weight if you take those pills? It’s highly unlikely. In case you can’t see the writing in the red box we highlighted, let us enlarge it for you:

You’ve seen enough infomercials to know that the vast majority are scams. They cherry pick extreme results and present them to you as truth. This is the exact same thing news outlets have done with Pokémon stories. The confusion stems from the fact that these stories are coming from reputable publications and sites, not infomercials that we inherently distrust.

So what about the analysis done by Revel Systems? Surely their statistics can confirm the “Pokémon Effect.” The problem with their analysis is that it was biased and lacked context.

When 63% of merchants reported an increase in sales, what was this in comparison to? Their study only included merchants in Chicago, San Francisco, and New York City - all of which see peak tourist volume in July (the month Pokémon Go was released). Merchants in these cities would have seen strong revenues regardless of the game’s presence. Below is a graph contrasting Revel System's figure with our estimate of non-PokeStops:

Amongst merchants in these cities that weren't PokéStops we found that 61% saw revenue gains. Given the sampling error that exists, the difference between PokéStops and non-PokéStops is statistically insignificant. Furthermore, Revel Systems indicated that merchants that did see gains saw an average increase of 12%. We found that the average merchant amongst non-PokéStops saw an 11.2% gain - once again, a statistically insignificant difference.

Why are we making such a fuss?

You may be thinking to yourself, “these are supposed to be fun analyses related to an interesting topic… why is Womply ruining the fun?”

To be clear, if one-off stories and infographics was where it all stopped, we would probably be playing along in the fun too. Unfortunately, everyone has taken Revel’s superficial analysis and begun making recommendations about how to drive business. Revel System’s CTO dangerously implied a $2,000 increase in weekly business for any merchant willing to participate in this new virtual world.

Additionally, highly respected news outlets, such as Forbes, CNBC, and inc.com, have all published articles about how to make money through augmented reality. These words will undoubtedly influence business owners moving forward.

Pokémon Go was the first of many augmented reality games, and its success exceeded anybody’s wildest imagination. Rest assured that game makers have taken notice and will capitalize on expectations the next time around. While players will likely remain the key revenue contributor for future games, there is no doubt that game makers will cash in (aka significantly increase prices) on the perception that game spots drive merchant revenue – and based on current prevailing perceptions, merchant owners will gladly pay.

Our goal isn’t to pick on the flaws of someone else’s analysis. We have every reason to believe that their statistics are factually correct. Our issue rests with how the data was interpreted: without context and implying causality. Unfortunately, with the way news is communicated and perpetuated today, the thoughts of a few misinformed can perpetuate throughout the world like a giant game of telephone at the speed of social media.

Channeling our inner Reinhold Niebuh, we hope that if you take away three things from this post, it would be that 1) There are often external economic factors you cannot change, 2) There are investments that will actually make a difference, and 3) Real statistics can grant you the wisdom to know the difference.

______

About Womply

Womply has a database of over 2 million merchants spanning 450 business verticals. Our partnerships with many of America’s largest credit card processors gives us a unique view into the spending patterns and behaviors of American consumers. Our proprietary data not only supports analyses like those seen in our blog posts and Statboard, but it also drives the analytics products that help SMB owners grow, protect, and simplify their businesses. For more information about us, check out our products or contact us at info@womply.com.

Pokémon Go - The Truth Behind the Game's Economic Impact

And just like that, it was over… 

News recently came out that Pokémon Go - the augmented reality app that had dominated global consciousness and phone screens since early July - had fallen from its perch atop Apple's App Store. This, coupled with the prior week’s news that the game's paying user base had dropped 79% since its peak, confirmed the game's rapid decline.

So what does this mean for small- and medium-sized businesses, many of which saw revenues skyrocket as a result of players loitering outside of their businesses? Contrary to most reports out there, not as much as you might think.

There’s no questioning the game’s impact

Pokemon Go was a global phenomenon that left an indelible impact. Below are just a few of the accolades the game boasted:

  • Most revenue grossed by a mobile game in its first month ($206.5 million in first month)
  • Most downloaded mobile game in its first month (130 million downloads)
  • Most countries in which mobile game was simultaneously ranked the most downloaded app (was #1 in 70 countries)
  • Most countries in which mobile game was simultaneously ranked the highest grossing app (was #1 in 55 countries)
  • Fastest mobile game to gross $100 million (a mere 20 days)

What may be even more impressive than these records was the way the game got people outside to experience actual sunlight (a concept that has become more and more foreign to each passing generation). With all of these players wandering the streets, news outlets began reporting on businesses seeing staggering increases in business.

Unfortunately there’s a difference between a few one-off observations and true causal impact. Were some businesses impacted? Absolutely. But merchants as a whole did not see revenues affected by the game.

Our super geeky analysis

(WARNING: readers who are allergic to math may wish to skip this section)

In the hopes of complementing the feel-good stories with hard statistics, our crack team of Data Scientists was locked in a room with a bottle of water, a bucket, and a significant sample of revenues from the company’s database of over 2 million merchants. We were promised great riches (aka new calculators and pocket protectors) if we could just put a number to Pokémon’s economic impact. Unfortunately, even this motivation was not enough for us to uncover anything meaningful.

Before we continue, it is important that we review the concept of hypothesis testing for the uninitiated. For those of you familiar with America’s judicial system, the notion of “innocent until proven guilty” should ring a bell. In statistics, there is a similar concept: “same until proven ‘probably not the same.’” This may sound like semantics to you - it may even sound stupid - but this is how statisticians think.

This means we start with the assumption that the revenue gains of merchants near PokéStops are the same as revenue gains of merchants not near PokéStops - holding all else constant - unless proven otherwise.

Another very important concept to know is random sampling. Samples are assumed to be somewhat representative of an entire group (dubbed the “population” in statistical parlance). This means that if you knowingly subset your data into a specific (i.e. non-random) group, you essentially negate any study’s result from being representative of the rest of the population.

So to recap, in order to statistically determine that PokéStops had an impact on merchants’ revenues, our task was to disprove that their revenue gains were the same as merchants not near PokéStops.

Our methods

WARNING: readers who are allergic to math will REALLY regret reading this section

Ceteris paribus

In Latin, ceteris paribus means "all other things being equal.” Our first task was to normalize the data so that all measurements of merchants’ performance could be compared “apples to apples.” Our analysis included adjustments for:

  • Annual seasonality trends
  • Day of week impact
  • Merchant size
  • Segment growth rates

Comparison of means

The most naïve way of comparing the impact is to measure the mean (aka “average”) percent change in revenue. To compute this, we evaluated merchants’ revenues during the six weeks prior to Pokémon’s launch and compared them to revenues after Pokémon’s launch. Using what is called a “Student’s t-test”, we were unable to note any difference between merchants near PokéStops and those not near one.

Comparison of distributions

A second way of analyzing the PokéStop merchants versus non-PokéStop merchants is to compare the distributions of their revenue changes. Just because the averages of their percent-revenue-change were the same does not guarantee that the samples still didn’t come from different populations. In the example below, we’ve depicted what it might look like if we were to sample from two different groups that happened to have the same average value:

 

In the example shown, the groups have different standard deviations, but they could theoretically exhibit any number of differences including skewedness, kurtosis, fat-tails, and several other funny sounding names.

Identifying differences in the distributions (such as the ones just listed) would be another indication that PokéStop and non-PokéStop merchants were indeed different. To do this we combined two methods.

First, we estimated the distribution of PokéStop and non-PokéStop merchant percent-revenue-changes using a method called Kernel Density Estimates (KDE). Below is the output of our sample distributions and density estimates:

You might be noting to yourself that the two distribution estimates aren’t exactly the same. Luckily, there is a statistical test to determine whether they are different enough to be considered meaningfully different. The Kolmogorov-Smirnov Test is a non-parametric test of the equality for one-dimensional probability distributions. Using this test, we can say that it is nearly impossible that these two samples come from different populations. To put another way, we still have no evidence that the PokéStop merchants and non-PokéStop merchants are different from one another.

Diffusion regression state-space model

If you’re still reading this, your commitment to understanding our methods is commendable. As a reward, you get to learn about the coolest sounding of all of our analyses: diffusion regression state-space modeling. This method utilizes Dynamic Time-Warping (DTW) to determine whether a latent variable (aka an unknown external factor) exists that could differentiate two time series. In this case, our latent variable was the potential impact of a nearby PokéStop (NOTE: if that made sense to you, we highly encourage you to go to our “Jobs” page and apply to a Data Science position).

Without getting into the gory details, we were once again unable to differentiate between PokéStop and non-PokéStop merchants.

Conclusion

Look, we didn’t set out to disagree with the world; we seek acceptance just like everyone else. In fact, we performed the analysis nearly two months ago with the hopes of being the first to publish a report on the game’s economic impact. When we came up empty, we figured there was no point publishing our results.

However, it was brought to our attention that other companies were communicating conclusions that we unequivocally disagreed with, and we were forced into responding. Womply’s mission is to use technology and data to grow, protect, and simplify small business, and we believe that poorly performed analyses have established a (potentially expensive) false belief about the impact of future augmented reality games.

The Pokémon craze may be over, but other augmented reality games are sure to come. Game developers will almost assuredly exploit merchants who believe that spending money on lures will increase business.

If this post was not enough to explain why we think merchants should ignore the hype (or at least proceed with extreme caution), our next post should quash any doubt about what you’ve read in the news.

______

About Womply

Womply has a database of over 2 million merchants spanning 450 business verticals. Our partnerships with many of America’s largest credit card processors gives us a unique view into the spending patterns and behaviors of American consumers. Our proprietary data not only supports analyses like those seen in our blog posts and Statboard, but it also drives the analytics products that help SMB owners grow, protect, and simplify their businesses. For more information about us, check out our products or contact us at info@womply.com.

Mother's Day vs. Father's Day - Restaurants

Welcome to post #2 of our Mother’s Day vs. Father’s Day analysis. If you took anything away from last week’s post, you’ll hopefully have learned two unmistakeable facts: 1) Moms love flowers, and 2) Dads hate flowers.

Shocking.

But this was just a microcosm of the broader notion that Mom’s and Dad’s simply like different things - at least when it comes to gifts and activities. How families dined out was a different story. 

Overall, restaurants saw a 47% increase in revenues on Mother’s Day and a 22% increase on Father’s Day. These are 2nd and 10th busiest days of the year for restaurants, respectively (Valentine’s Day was the runaway winner with 68% revenue gains). But while the volume was different, the behavior converged significantly.

To Splurge or Not to Splurge… That Is the Question

How do Americans pick where to eat to honor Mom’s and Dad’s? If you’re a wine expert, the following analogy won’t make much sense to you. For the rest of us mere mortals, this will probably sound awfully familiar.

Think about the last time you had to go to a housewarming party and you had to bring a bottle of wine. How do you pick? Maybe you think “Napa” or “France” and go from there... but more often than not you’re going to ask yourself, “how much should I spend so that they’ll appreciate the gesture (after they inevitably Google how much it cost), but so that I don’t break the bank?

Well it seems as though Americans have taken a similar approach when it comes to dining out. The figure below shows the average spending increase on the two holidays for each price range (price range being defined as the average price index amongst popular online review sites):

While restaurants of all price ranges saw meaningful gains on the two holidays, there was a clear choice when it came to the most preferred. At 91% revenue gains, $$$ restaurants saw twice the increases seen by their $$ and $$$$ counterparts. This was consistent for both Mother’s Day as well as Father’s Day (albeit at different magnitudes). Clearly families wanted to splurge, but not splurge too much.

Cuisine Style

When it came to cuisine style the two holidays, we once again saw significant agreement with seven of the top 10 options overlapping. Mediterranean, Steakhouses, Italian, Fine Dining, Japanese, Indian, and Seafood restaurants were all top choices on both Mother’s Day and Father’s Day. 

The worst performing Mother’s Day restaurants were predictable. Chicken Restaurants, Taco Restaurants, Burger Restaurants, and Hot Dog Restaurants all suffered significant declines compared to their more exotic (a.k.a. fancier) options. 

Families were far less discerning on Father’s Day, with no particular cuisine style seeing significant declines, with the exception of one - Vietnamese Restaurants.

Wrap-up

While always interesting to see how American consumer behaviors unfold, none of our findings here were entirely unexpected. 

Families spent more than twice as much celebrating Mother’s Day as compared to Father’s Day.  Mothers know best. Their knowledge, nurturing, and love shapes lives. They are the emotional center of most families. As Dorothy, from the In the 80’s sitcom The Golden Girls, put it, “It's not easy being a mother. If it were easy, fathers would do it.”

“It's not easy being a mother. If it were easy, fathers would do it.”

-Dorothy, from the In the 80’s sitcom The Golden Girls

Ultimately, whether it’s a family with two parents, a single mom, a single dad, grandparents, step-parents, god-parents, or guardians, these are the people in our lives who influence us like nobody else. They deserve all the recognition in the world, and if spending is a good indicator, it looks like Americans are pretty good at giving them exactly what Mothers and Fathers asked for.

______

About Womply

Womply has a database of over 2 million merchants spanning 450 business verticals. Our partnerships with many of America’s largest credit card processors gives us a unique view into the spending patterns and behaviors of American consumers. Our proprietary data not only supports analyses like those seen in our blog posts and Statboard, but it also drives the analytics products that help SMB owners grow, protect, and simplify their businesses. For more information about us, check out our products or contact us at info@womply.com.

Mother's Day vs. Father's Day - Non-Restaurants

On average, spending on Mother’s Day festivities more than doubled that of Father’s Day. This should come as little surprise given the National Retail Federation (NRF) annual survey, which highlights the spending gap between Mother’s Day gifts and Father’s Day gifts.

And while the conclusion of this study matched that of the NRF, our analyses differed significantly. Rather than consumer surveys, we analyzed actual spending observed on the two holidays. Thus, you will see results focused heavily on restaurants, last-minute gifts, and specialty services and activities.

Non-Restaurant Spending

Mother’s Day

Below is a chart of the top 10 non-restaurant categories:

fig1.1.png

With the exception of Farmers Markets, there were no real surprises about the composition of the top 10 merchant sub-categories. At an average revenue increase of 197%, Flower and Gift shops were far and away the busiest, with Candy and Dessert Shops (75%) and Jewelry Stores (50%) completing the trifecta of cliche last-minute offerings. By comparison, Valentine’s Day this year saw 275% gains for flowers, 168% for candy, and 102% for jewelry.

What we did find surprising was that Day Spas, Massage Parlors, and Salons did not perform as well as expected. While still top-10 performers, these merchants were expected to see revenue gains exceeding 50%. Our hypothesis is likely that these businesses were indeed busy, but that a large percentage of services were paid using gift certificates (where revenue is booked at the time of certificate purchase).

Perhaps just as interesting is a look into where money wasn’t spent. Below is the bottom ten performers in the non-restaurant category.

Eight of the 10 worst performing business types were sub-categories more frequently associated with men. As the saying goes, “Happy wife, happy life,” and it looks like America’s men knew where to avoid on Mother’s Day.

Father’s Day

Below is a chart of the top 5 non-restaurant categories on Father’s Day

fig1.3.png

Predicting what would comprise the top Father’s Day categories was far more difficult than for their maternal counterparts. The two categories that nearly doubled their expected sales - Hunting and Fishing Stores (94% increase) and Guns and Ammo Shops (91% increase) - are highly influenced by geographic region.

At 34% increase, Golf Courses were the next sub-category most improved by Father’s Day. Like spas on Mother’s Day, this figure seems lower than we originally expected. However, given that 1) courses are already fairly busy in the summer, and 2) courses have finite tee times (thereby limiting how much excess demand they can serve), it makes sense that golf courses didn’t outperform other sub-categories as much as we had expected.

Theme parks (which are highly influenced by the start of summer, irrespective of it being a holiday) and Sporting Goods Stores were the only other sub-categories that saw significant increases on Father’s Day.

Meanwhile, below are the bottom 10 sub-categories:

Mirroring what we saw for Mother’s Day, the majority of poorly performing sub-categories were those typically associated with the opposite sex. Furthermore, Flower and Gift Shops were the runaway losers on Father’s Day. Apparently nothing says “I love you, Dad” like not buying flowers.

Comparison

Seeing gender normatives and stereotypes play out is obviously fun, but what we found most interesting was the underlying implications of these spending behaviors. 

Amongst the top-10 sub-categories on Mother’s Day were:

  1. Hair Salons
  2. Massage Parlors
  3. Day Spas
  4. Nail Salons
  5. Yoga and Pilates Centers

While one could argue that getting nails or hair done could be mother-daughter activities, these are generally activities done in isolation. Mother’s Day often means a few hours of solace from the hecticness of everyday life. In fact, a recent survey found that 71% of mothers wanted some “me time” to celebrate their special day. These results confirm this sentiment.

This stands in stark contrast to the Zagat survey that found that 52% of dads just wanted to stay at home and relax with the family. Our analysis confirmed this sentiment as well, with Golf Courses being the only sub-category to see significant revenue gains on Father’s Day. In fact, when looking just beyond the bottom-10 for Father’s Day, we saw sub-categories like Sports and Activity Places (-10%), Bowling Alleys (-10%), and Pool Halls (-8%) - all activities more typically associated with men being away from the family - all seeing revenue suffer on Father’s Day.

Up Next: Part 2 of our Mother’s Day vs Father’s Day analysis takes a look at Americans’ spending behavior at Restaurants.

______

About Womply

Womply has a database of over 2 million merchants spanning 450 business verticals. Our partnerships with many of America’s largest credit card processors gives us an unique view into the spending patterns and behaviors of American consumers. Our proprietary data not only supports analyses like those seen in our blog posts and Statboard, but it also drives the analytics products that help SMB owners grow, protect, and simplify their businesses. For more information about us, check out our products or contact us at info@womply.com.

Cinco de Mayo - A Case Study in American's Love Affair With Celebrations

Cinco de Mayo is not Mexico’s Independence Day. 

Surprised? That’d put you in a group with the large number of Americans who have come to know the date as, “Drinko de Mayo” or “Cinco de Drinko.” (NOTE: Cinco de Mayo actually commemorates a battle won in 1862 by undermanned and under-equipped Mexican forces over the mighty French Army).

Ranking 48th in the world in per capita alcohol consumption, Americans like drinking. They don't love it… at least when compared to the true drinkers of the world (Belarus drinks 90% more per capita than the U.S.!), but it's certainly a huge part of the culture.

What Americans do love, however, is having a reason to drink. They will pretty much celebrate anything (check out the rules to this Arbor Day drinking game). So when beer companies came up with the idea in the 1980s to push Cinco de Mayo as a celebration in the U.S., Americans flocked to it like ants on a fallen ice cream cone.

So how do Americans commemorate this all-important battle of the Franco-Mexican War? By going to Mexican Restaurants, obviously. In fact, the holiday boosted typical revenues of Mexican Restaurants by a whopping 140% this year.

So would any Mexican Restaurant suffice? The graph below shows the revenue increases of Mexican Restaurants, broken down by the availability of alcohol:

cinco_de_mayo.png

Clearly just serving up quesadillas and tacos was enough to drive significant gains. Mexican Restaurants serving no alcohol saw a 75% increase in revenues.

Meanwhile, burritos and beer was an ever greater boon. Mexican Restaurants serving beer (i.e. no hard alcohol) saw their revenues increase by 110%.

The real winners, however, were the full bar Mexican Restaurants. These establishments saw their business nearly triple on Cinco de Mayo, with 197% revenue gains.

So to all of the associations out there promoting National Fried Chicken Day, Donut Day, or any of the other “Day” meant to promote businesses in your category, I wish you the best of luck. Just remember, if you’re ever finding that you need a little extra boost to get your holiday over the top, I have two words for you….. Tequila shots.

______

About Womply

Womply has a database of over 2 million merchants spanning 450 business verticals. Our partnerships with many of America’s largest credit card processors gives us an unique view into the spending patterns and behaviors of American consumers. Our proprietary data not only supports analyses like those seen in our blog posts and Statboard, but it also drives the analytics products that help SMB owners grow, protect, and simplify their businesses. For more information about us, check out our products or contact us at info@womply.com.