Game Analytics I: DON’T Measure Everything

It’s no secret that turning a profit in mobile games isn’t easy these days, and not only because competition is fierce. The modern mobile game product is an intimidating construct with many moving parts—code, staged releases, content production, backend, 3rd-party integrations, liveops schedule, QA, device and OS fragmentation, community management, customer support, and marketing and distribution just to name a few. When managing such a product, it’s easy to get swept up by the hurricane of inputs and lose sight of the big picture: where we are, where we’re going, and what deserves our focus right now.

As a freelancer and consultant focused on game Product Management, many of the questions I hear are permutations of same basic question, “How should we (better) use our data to make decisions?” For example:

“What should we be focusing on?”

“How can we find the small adjustments that yield large returns?”

“How can we improve our playbook, leveraging data effectively and consistently across the company?”

These questions implicitly reflect the challenge of making analytics, statistics and game business models easy to understand. They also hint at why a competitive industry sometimes (quite innocently) incentivizes doing the opposite.

One of the most important lessons that I’ve learned (and relearned) over the years is the miraculous power of keeping things as simple as possible. This principle has countless applications, but has become particularly fundamental to how I approach mobile game design, analytics, and live operations.

After six years in mobile games production and product management, and after making and learning from, let’s just say “plenty” of mistakes myself, I’ve found that a common-sense and simple approach to game telemetry nearly always yields the greatest returns. As enticing as it sounds to simply “track everything” that users do within our game or mobile app, my experience has been that when rubber meets the road, this approach fails for six reasons:

  • Bugs – All other things equal, tracking more actions leaves less time to implement each. This tends to increase bug counts.
  • Loss of Trust in Data – Given bugs, without 100% confidence in and familiarity with the data you can’t quite trust your results and conclusions. You’ll find yourself cross-checking with other data, only to find another data point that seems off, and down the rabbit hole you go…
  • Visual and mental clutter – You may find yourself routinely wading through long lists of superfluous metrics to reach the important ones.
  • Not a substitute for planning – A shotgun approach can be a warning sign that the data design was created without consideration for how the data will be used.
  • Encourages bad habits – The added noise and clutter tempts us to tunnel in on metrics and hypotheses that aren’t actionable.
  • Makes you a bottleneck – Additional complexity makes it harder for others on your team to do independent analysis.

For these reasons and others, I prefer to start by establishing a rock-solid foundation that relies on only tracking six types of user behavior. The next post in this series will explore this topic in further detail.

Getting back to the bigger picture, this almost certainly won’t be your first encounter with the “keep it simple” platitude. But – perhaps there’s value in a tactical, “boots on the ground” discussion on how a minimalistic and focused approach to Product Management can serve us at multiple levels—from the very bottom (the specific list of game events and hooks implemented in code) to the top (the language used to discuss performance with the team and CEO).

Given that, the articles that follow will explore, in more depth, how to apply a simple, common-sense approaches to mobile game telemetry and data-driven decision-making. Of course, there’s no one right approach. Every mobile business has a unique perspective and toolkit, and my favorite thing about consulting is the learning and knowledge-sharing that comes from these experiences.

The next article: ‘Just track these six behaviors’ dives into tactical practice, and will be most relevant to mobile product managers already familiar with the basics of mobile game telemetry.

But, no matter who you are, I’d love to hear your thoughts and perspective. Don’t hesitate to drop me a line at [email protected].

This series: ‘Mobile Game Analytics’ will continue with Part II: Just track these six behaviors.

About the Author