Every marketing experiment has to have a solid hypothesis – unless you want your experimentation program to go astray. Speaking at the Netcore Martech Mashup 3.0, Varun Rama Murthy, Group VP, and Business Head, Netcore, time travels to the early 2010s – to his Zynga days – to recall how the company framed effective hypotheses.
What is a hypothesis and how do I effectively frame one? While this question may seem straight out of a scientific journal, marketers are increasingly turning to this empirical method to boost digital results and squeeze the most out of their online campaigns.
Every marketing experiment has to have a solid hypothesis – unless you want to be fumbling in the dark with your experimentation program.
From a product management standpoint, a hypothesis is an idea that is observable but not proven yet. Its motive is to help gain insights, regardless of the testing outcome. Also, a hypothesis can’t be pulled out like a rabbit from a magician’s hat. It has to be strictly based on research data.
How does one create an analytical hypothesis? Here’s a textbook guide to building a valid hypothesis:
- Observe: Analytics is a rich source of observations.
- Ideate: Based on your observations, you come up with ideas.
- Weed: Remove all the ideas that cannot become a hypothesis.
- Frame: From there on, frame your hypothesis.
Let’s zoom in to get a micro view of each step.
Step 1: Observation
Observations are nothing but what you spot and perceive. Product managers often think that observations come only from analytics. But there are a lot of places from where observational data can be collected for formulating a hypothesis.
- Internal Data: Besides analytics, internal data that can be effectively observed includes customer tickets, reviews, and surveys.
- External Data: This makes interesting fodder for observation. For instance, a competitor’s product launch can provide you some compelling metrics to observe. You can also observe proxies. Let’s say you’re designing a game economy. Then you could look at how the US functioned after the massive liquidity crisis and use that as a proxy for figuring your game economy.
- Trends: New innovations and customer behaviour. For example, when Apple came out with the iPhone, it set off a new trend. Consumers started looking forward to the new smartphone market behaviour. That could be a basis for an observation.
Cut to 2012, when Zygana launched YoVille (now officially renamed YoWorld), a game of self-expression. As a first step to formulating a hypothesis, the company observed a bunch of things, using both internal data and metrics from outside Zynga.
What were Zynga’s internal-data triggered observations?
- Repeat purchases were on the ascent. People were buying their second, third, fourth, and even fifth items.
- There was a predictable frequency with which repeat buyers made purchases. Zynga identified the timeline as a two week cycle between the second, third, and fourth purchase.
- Drilling down further, the gaming company observed that there was an 80/20 split in the type of purchases made. YoVille sold two types of virtual goods – clothing and furniture. People who bought clothing never bought furniture, and vice versa.
- The Average Revenue Per Paying User (ARPPU) was going up. That was great, or so Zynga thought.
- On closer look, the company realized that the numbers of repeat buyers was decreasing. So while the denominator was dipping, the ARPPU was on an ascent.
- Along with all this, there was constant clamour on the forum, as players kept asking for cheaper goods.
Step 2: Ideation
Ideas are not equal to a hypothesis – they are more like educated guesses. Ideas generally tend to start as questions, like, “This is an observation, and how do I grow it further?” Ideas may or may not be testable.
Going back to Zynga, 2012. The company was facing a bunch of issues, like the discord between its ARPPU and paid users. Zynga’s think-tank went into a huddle, and using its observation sets, it put five ideas on the table:
- Product bundling: What if we bundle a few products together and sell it at a certain price?
- Higher frequency of content drops: Instead of dropping content every two weeks, what if we do it every one week?
- Freebies: Why don’t we give free furniture to our clothing buyers and free clothing to our furniture buyers?
- Subscription driven payments: Why don’t we go for paper-driven subscriptions?
- Let’s do nothing: What if we’re witnessing a regular games cycle? Most games have a sharp rise in user base, before they start coming down. The downward trajectory is usually not as rapid as the upward one. So what if we do nothing, since this is the nature of the game?
Step 3: Weeding
Weed out ideas that are not hypothesis. There are three things that make an idea a hypothesis:
- It has to specifically impact a metric. If you do something, it has to have a measurable, black-and-white metric.
- It has to be testable. You should be able to run a test to figure out the validity of the idea.
- Falsifiability. If an idea can be disproved, just like it can be proved, it is a falsifiable idea. For instance, a marketer decides to revamp his company homepage to impact conversions. Is the idea specific? Yes. Is it testable? Yes. Is it falsifiable? No, because the statement cannot be disproved. This is the place where marketers don’t want to be – where you can’t prove or disprove an idea.
Using the weeding process, Zynga was able to narrow down to four of its five ideas (that were both testable and falsifiable), and started generating a hypothesis around them.
Step 4: Framing a Hypothesis
What should an ideal hypothesis in the product world contain? A good product hypothesis is made up of three components, or the MIA Framework:
- Modification: What are the new elements that you are bringing in? These could be new experiences, products, or changes.
- Impact: What will be the probable impact of this new element? This can be called the most important part of the hypothesis statement, and is also something that marketers tend to ignore.
- Audience: Who are you suggesting this change to? This refers to the population sample that will be subject to the modification.
Using these reference points, the Zynga hypothesis looked something like this:
- Introducing a recurring payment model became its key Modification.
- The Impact it created was a 5% increase in ARPPU and a 1% increase in retained payers.
- The Audience that Zynga targeted was its overall paying population.
So what happened with Zynga’s hypothesis (of subscription giving a 5% jump on ARPPU) in a real world scenario? Zynga tested the hypothesis for two months and discovered that it was proven right, and the actual ARPPU impact was 12% on the baseline while the net payer retention was 2.5% on top of the baseline.
Closing Thoughts
Hypothesis is the foundational bedrock of product management. However, some of the things that product managers should keep in focus while designing a sound hypothesis include:
- The constraints around a hypothesis: These include cannibalization (Would a subscription use case cannibalize other revenue?), timing (Is this the right time to run this hypothesis?), and execution.
- Exit strategy: Always have a fail-safe. If your hypothesis doesn’t work, you might be going back on a promise made to your customers. So always have an exit strategy in case a hypothesis fails.
- None of this is possible without a relevant organizational culture: Have a culture built around hypothesis and experimentation.
