To hijack the title from a movie from 1963, anyone involved in building products or services today, especially Software as a Service (SaaS) apps, live in a mad, mad, mad, mad world. SaaS projects are all about time-to-market because there is a perceived first-mover advantage. The race is on from the start of the project to the first preview and the pedal never seems to come off the metal—shipping every week is the norm. Besides the crazy pace, the complexity is very high—how to work in a agile fashion when there are many moving parts (build trains, scalable server farms, partner teams who aren’t agile), and being confused by different versions of the truth (micro-feature changes slip-streaming into production, split testing, etc.) such that it’s hard to get a valid and reliable take on what customers are doing and how satisfied they are. What to do?

The SFE Group has seen this sort of scenario play out on many teams: Susan, a product manager, is totally smitten by web analytics and other “data exhaust” sources. She is amazed by the amount of information about customers she has access to in the SaaS world, and the near-real-time nature of that information. And, her team members all genuinely care about doing the right thing for customers, unlike 5-10 years ago when they were “too busy” to find out what customers were up to. However, Susan starts to wonder if there is something more she should do (that’s not too hard or expensive) that will help her more deeply understand the meaning behind these data sources, and thus enable her to make better decisions about what to fix or build next, how to prioritize new work, and how to handle the anticipated customer requests…

The short answer is: Yes – it’s fairly easy these days to gather reams of data to help you get a deeper understanding of your customers! However, the long answer is: the easiest data to collect is also often the least valuable.

Consider the broad array of types of data that can be collected:

Two of the most-common sources of data available to SaaS teams are weblogs / instrumentation and A/B testing results, which are in the upper right quadrant (QUANT DO data), and data from closed-ended survey questions, which is in the upper left quadrant (QUANT SAY data). Why is that a problem? The axes give a clue—those data sources tell us what customers are doing and saying but not why. Understanding WHY is crucial to extracting more than superficial learnings from data. Your customers’ needs and problems are often nuanced, so you need to demand more of your data than just piles of history around what they have done. You need wrestle with diverse data sets – make it give up its insights, so that you can make better-informed decisions. We chose to use an elephant for this chart to reference the blind men and elephant parable. In case you’re not familiar, “a group of blind men (or men in the dark) touch an elephant to learn what it is like. Each one feels a different part, but only one part, such as the side or the tusk. They then compare notes and learn that they are in complete disagreement.” The parable nicely illustrates the importance of getting multiple perspectives and comparing them, to produce a more complete understanding.

Large volumes of data, especially numbers, have gravitas, so it’s especially easy to be seduced into thinking those types of data sources are giving you all you need. And analysis of those large data sets often plays well in upper level management presentations. Yet we know that these data sets don’t tell a complete story and many times don’t help you understand why a customer behaves the way they do. So why do large numerical data sets have such high “face validity”? Our brains are meaning-making machines: We unconsciously infer other peoples’ motives and reasons and implicitly substitute our own assumptions. The trouble is, sometimes we’re dead wrong about why people do the things they do. We need to be aware of this blind spot and be intentional about how many quadrants we bring to bear for a given question. At minimum, we should always balance numbers with understand why customers do the things they do.

Therefore, in a world where the complexity is high and the pace is insane, it’s only natural that teams gravitate to data that is easy to collect and that provide “safety in numbers”. But, just like a sugary breakfast cereal, it’s (at best) only a part of a well-balanced data diet. You and your team need to make time, and if necessary, bring on people with expertise, to collect data on the left side of the graph, to provide the crucial, missing why. In a future blog post, I’ll talk more about how to do that in a SaaS world.