As a consultant, I’ve had the opportunity
to learn from marketing teams about marketing
tactics that either did not meet expectations
or failed completely. To give you an example, a bank wanted to increase
their customers’ debit card usage, so they
decided to roll out a rewards program for
all their customers. They also invested heavily in advertising
about the program, but six months down the line they realized that the program was not
really delivering any incremental revenues
or profits for the debit cards. They were not clear if the program worked
or if it worked for only a specific set of
customers or if it worked, then how big was
the size of impact? It is surprising that even in today’s data
rich world there exists a gap between what
businesses think would work for their customers
versus what actually works for their customers. And this is quite evident from the low marketing
response rates that we typically see.
This gap is only going to get bigger because
in today’s world, customer behavior is evolving rapidly. What’s the solution?
The only way to know what actually works for
your customers is by doing an experiment. Many leading organizations have realized that
experimentation is a great way to empirically
vaildate your ideas by collecting actual customer
response data at a much lower investment. Once your experiment succeeds, you can go
ahead with a full scale high investment rollout
with higher confidence. Going back to the banking example, had this set up an experiment first, they would know exactly which customers respond and which customers don’t and the ones that don’t respond is in fact
a wasted investment. And additionally they would also know how big the impact was because they can now baseline it to a group of control customers who didn’t
get the rewards.
So do businesses experiment? Yes, a lot of businesses do experiment, but
they’re doing it in an ad hoc way.
There are three challenges that they face
in terms of institutionalizing the process of experimentation. The first challenge is lack of good design
principals. Every customer is unique.
Every retail store is unique.Every bank branch is unique. So the science behind choosing the right set of test and control subjects is quite complex and businesses may not have the right skills to do it in house. The second challenge is inconsistent measurement
approaches leading to inconsistent results,
which lead to more discussions and debates
rather than answers. Additionally, it makes cross pollination of
ideas and learnings across teams and across
markets extremely difficult. The third challenge is expensive methodology
which is difficult to scale.
Most businesses use a manual way of analyzing
experiments When we are looking at a global organization
with multiple brands across multiple countries,
we are talking about a high volume of experiments
and this process starts to break down. So how can you set up a culture of experimentation?
Make it easy to test ideas.
Everybody in the organization should be able
to set up an experiment and analyze it easily. That’s the only way to drive higher adoption.
Two, make it a part of your KPI’s. Everybody in the organization should actively
seek out ideas to test and subsequently celebrate
successes. Three, set up experimentation champions.
People who understand the statistical complexity
of the process at the back end and who can ensure that the sanctity of the process is
So accelerating experimentation will provide
the right boost to your business in terms
of increasing your returns on investment.