Why is experimentation so important in digital marketing?

You’ve probably heard the term “Experimentation” more and more lately. But perhaps you’re not completely aware of how important it is now and will be in the future.

In my opinion, experimentation is the most important concept in business today. Even Avinash Kaushik, a leader in Digital Analytics, recently said:

EXPERIMENTATION HAS THE POWER TO OVERSHADOW THE IMPACT YOU CAN HAVE WITH DIGITAL ANALYTICS!

— Avinash Kaushik (Digital Analytics Evangelist)

Popularity of conversion rate optimization has constantly been growing.

Google Trends shows that the popularity of searches for this topic has basically doubled since 2015:

As someone working with CRO for 10 years, I've also noticed how this topic is becoming more and more mainstream.

It used to be just a weird name for a few nerds trying to replicate the scientific method in digital interface design processes.

Now, it's a critical best practice for marketers, software engineers, and product people, among others.

Another way to gauge CRO’s popularity is by looking at the thousands of job opportunities available on Linkedin:

Impressive for such a specific area that kind of didn't exist 10 years ago, right?

Why is CRO rising in popularity?

For starters, it’s important to clarify that whoever explores experimentation correctly will be able to dominate their market (like some have already done). While whoever brushes it off as “just another hype” and ignores it, will end up being left in the dust.

The CRO industry is growing for a number of reasons. One of them is the fact it works, of course.

But one more explanation for that constant growth is that everyone wants to be the next Google, Amazon, Microsoft, Netflix, Spotify, Uber, or virtually any major tech industry leader nowadays.

And they are all known for investing a lot in CRO and Experimentation, running tests on everything they can.

Stefan Thomke from Harvard compared the stock performance of companies known to invest heavily in Experimentation with the S&P 500:

Source: Experimentation Works, 2020

Even though it's evident that these companies didn't grow only because they run AB tests, it's fair to say that Experimentation is at the core of their culture and processes.

Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.

-- Jeff Bezos (Founder), Amazon

And who wouldn't want a performance like theirs, right?

There are too many people calling anything an “Experiment”

Not a day goes by without someone going around saying they ran an “Experiment” (or a “test”), when in fact, even unknowingly, they didn’t even come close.

If you created something new or explored a new strategy to improve your current results, it is mandatory that you do A/B Testing in order to call it an “Experiment”. Otherwise, all you’ve done is created something new, not having earned the right to use the terms above. 🥲

(Note: There are various experiment/testing formats. To simplify things, in this article we will refer to the simplest of all — A/B Testing.)

A/B Testing is referred to as a Randomized Controlled Experiment (or Trial).

1. Group A - The “Control” group is the Original version without modifications; it's NOT exposed to whatever you are changing
2. Group B - The "Treatment" group is the Experimental version with modifications; it IS exposed to whatever you are changing

This way, you isolate all variables and accurately measure the effect caused by what you are testing. In other words, the Control and Experimental groups go head to head in the same conditions, for audiences that have the same profile, during the same time period, and under the same external influences, etc.

I don’t want to undermine the efforts of those who execute actions that don’t go through testing. In some cases, there are indeed ways to try to understand the real, approximate impact caused by them, after the fact. But they all pale in comparison to experimentation — both on matters of reliability and accuracy.

Ron Kohavi, a scientist who played an important role in the experimentation programs in companies like Amazon, Microsoft, and Airbnb, defines A/B Testing (Randomized Controlled Experiments) as the “gold standard” required to attribute causality to something. In other words, to understand the impact caused by an element or action.

Notice that, according to science, there is profound vulnerability in approaches like “studies done by specialists,” which many companies consider to be enough to understand phenomenons. This kind of approach carries great risk and confusion between correlation and causality. 

Yes. Analyzing isolated data is not enough to draw safe conclusions on cause and effect. Want to see why?

Take a look at this graph from 1949. It shows the correlation between ice cream sales in the US and the number of polio cases:

Source: Axbit.com, 2019

Interesting, right? I know. According to the data analyzed on this graph, health professionals asked people to stop buying ice cream in order to reduce the number of polio cases.

Naturally, in time it was discovered that there was no causal relationship between the two pieces of data. The spread of polio was more common in higher temperatures — during warmer months in the year — just like ice cream sales.

Did you find this example too far-fetched? Even today, we often see marketing professionals attributing a product’s decrease in churn to the use of a certain feature without doing reliable experimentation to actually test the theory.

It is the same exact mistake, more than 70 years later: to suppose that there is causality where possibly only correlation exists.

Why the best companies invest in Experimentation

Think about any digital company that is a world leader in an extremely competitive industry.

It is very, VERY likely that you thought of a company that is obsessed with experimentation — a place where everything that can be tested will be. More than that, the entire company embraces and encourages the highest number of experiments to be done.

Airbnb, Amazon, Booking, Facebook, Linkedin, Netflix, Spotify, Uber …

Go ahead and look into how innovation and decision making works in each one of these companies and in other key players today.

Let’s go over some of the reasons why these companies take experimentation so seriously (and why you should, too).

1. Experiments save you from making potentially expensive mistakes

No matter how good you are, there will be times when your predictions about the outcome of a certain action in your company will be wrong. 

It also doesn’t matter how good your research or discovery were when justifying a step you want to take. Real chances are: things may still go wrong. 

Throughout the years, I’ve witnessed various companies (some of them, excellent) run into trouble for making changes that looked like “obvious” or very well-founded improvements, but that turned out to be steps backwards. 

💡 Did you know that only about 10% of tests done in brilliant companies such as Google and Microsoft succeed? 

Now, can you imagine if these companies made these changes without testing them? Have you thought of how much time and money would be lost with 90% of their efforts actually setting the company back?

One of the great beauties of experimentation is that it keeps you from implementing changes that will set you back (damage your KPIs). The worst that can happen is your test fails and you keep performing at the same level.

2. Experiments accelerate your innovation

Let’s take a moment to crush a gigantic (and very common) myth about experimentation.

Many people believe that running experiments will slow your company down. 

"Put everything through A/B Testing??? Impossible. It’d make us too slow! That wouldn’t work here. We’re too dynamic."

Sounds familiar, right?

I understand and sympathize with this, it’s a valid concern. After all, in the digital economy, if you don’t grow quickly, you get crushed by the competition.

Truth is, before any change goes live for good, you do need to place it in a testing environment until enough users have viewed it to reach statistical significance. This can take a couple of days, a few weeks, a month, or even longer.

However, when you actually adopt this process, the time you spend waiting for statistical significance is more than compensated by outcomes such as:

• Experimentation accelerates decision making. The certainty that an initiative will go through reliable validation makes it so there are dramatically less discussions and bureaucracy keeping it from coming alive.
• Measuring the results of an effort is much easier. When there is no experimentation, it is common to see companies interrupting the work of senior professionals and even entire teams to try to understand the real impact that resulted from a change that was made recently. In A/B Testing, accurate results are accessible to everyone.
• Lastly, one of my favorites: With experimentation, you don’t have to implement every new thing you do with the highest-quality code, translated into every language, perfectly responsive in all devices, with complete documentation, etc. While testing, you are concerned with implementing the minimum necessary to validate your hypothesis. You will only invest in everything you need for a final delivery if the experiments show that your hypothesis truly works. This can drastically increase a company’s efficiency and productivity.

3. Experiments democratize your innovation

When a company has a good experimentation culture, there are less obstacles keeping people from innovating. With that, more people will try to innovate through experimentation. And the more people experiment in their companies… Well, as Jeff Bezos put it in the beginning of this article: more experiments = more growth.

This is why the most innovative companies are so concerned with creating an environment where everyone can run their experiments.

Something interesting happened at Booking.com, where around 75% of their thousands of employees in the technology and product fields actively use the company’s experimentation tool.

In December 2017, a critical month for websites in their industry, a designer proposed a test with a brutal redesign of the homepage. The various elements at the top of the page, which the company had spent years optimizing — lists of accommodations, deals, photos, texts… — would be replaced by a simple form asking the user where they wanted to go, when, and with how many people.

The CEO of the company and other directors doubted that the idea would work. They went as far as betting a bottle of champagne on it, believing the conversion rate would drop. However, the experiment was still authorized to run.

After all, they were “all in” on the idea of developing a culture of experimentation and democratizing it as much as possible. The agreement was that anyone could test anything they wanted, without having to get authorization from the higher-ups.

(What do you think? At your company, would a test in a similar context go live?)

This is full-scale innovation, where everyone participates through experimentation.

Oh, you want to know how that crazy experiment turned out? Well… all you have to do is take a peek at Booking.com’s homepage today, years later, to find out. 🙂

4. Experimentation is the best way to get to know your customer

Don’t get me wrong. I love (and use) techniques such as:

• Product discovery
• User interviews
• Usability tests
• Searching for opportunities in Analytics tools
• Audience surveys

But none of them have as much accuracy and reliability as an experiment does.

Let me illustrate: A few years ago, Intuit, a financial management software for companies, held 20 interviews with users to know if they would be interested in a faster way to pay their employees when they created accounts. The interviews came out with a hard no. Unanimously.

However, in an A/B test, the option to pay quicker performed better and turned out to be excellent in the long run, consistently increasing Intuit’s revenue.

There will always be differences between what people say, what experts understand from what they said, and what they effectively do.

Experimentation focuses on the last item. It evaluates — without any interference — what your user genuinely prefers, what makes them stay longer, what makes them come back, what makes them spend more, etc.

At each test result, whether it’s positive or negative, you gain valuable understanding about your customer.

Experiments are a truth serum getting injected directly into your customers’ veins. 👀

That’s why this popular principle rings so true:

A visitor who doesn’t go through A/B Testing is a learning opportunity gone to waste.

Those who truly want to dominate their market take this principle to heart. That’s why, according to some internal information, every time you go to your Facebook, there is a 99% chance you’re participating in at least 10 experiments.

How do I start to implement experimentation in my company?

Great question. There is much to be said about how to implement and develop experimentation in a company. I will touch on a lot of related content from time to time at Seer's blog. But for now, let’s focus on a few important points you need to know.

The way I see it, in order to execute experimentation at a high level, a company needs to take good care of these 3 pillars: people, technology, and culture.

People

Unfortunately, experimentation is not the most trivial topic in the world, and implementing it with technical errors can result in extremely expensive mistakes since the decisions being made at the head of the company are led by test results.

This is why it is essential to surround yourself with people who are able to lead your experimentation efforts the right way.

At the same time, as you’ve seen in this article, experimentation only reaches its highest potential when it is embraced by the entire company, when all teams run experiments.

This is why the organizational model that I see working best in companies that are excellent at this is a mix of the centralized and the horizontal structure:

In this model, there is a core experimentation team that centralizes the overall responsibility for the proper functioning of the experimentation program: process creation, technology, evangelist efforts, communication, etc. This team can be a third-party specialized team, or, if you already have the right in-house talent, you can assemble them in this centralized team.

The core team should provide the necessary training and support for all other teams/squads in your company to apply experimentation.

Keep in mind that this team may not even do their own A/B Testing. That’s not their main role. They exist so that the rest of the company can run their own testing.

One of the coolest things about this structure is that it allows for scaling and avoids the pitfall of believing that every team in your company will be able to — of their own volition — effectively inject experimentation into the heart of their processes.

(If you’d like me to go deeper into the topic of people / organizational structure in future articles, speak up 🙂)

Technology

If you want to obtain reliable results from testing and to scale your experimentation program, you’ll need to pay close attention to your experimentation platform.

Your technological needs will depend greatly on the level of maturity you find yourself in. But in terms of technology, you will face questions such as:

• Should I hire an external tool or create my own?
• How do I integrate my business data with the testing tool? Many popular A/B Testing tools (‘sup, Google Optimize!) make it hard or impossible to meet the needs of various companies in this regard.
• How do I do server-side testing?
• How do I run simultaneous tests in a reliable way?

Remember: If your platform supplies you with inaccurate results, you may potentially doom the future of your entire company with wrong decisions. If it doesn’t allow you to be quick, you may become an easy prey to your competition.

For this and other reasons, don’t underestimate the need to set up your environment with solid technology.

Culture

A broad term for a broad topic. Experimentation culture involves various factors — and nothing works without it.

For instance, your company needs to be truly data-driven. Your testing results must be taken seriously and speak much louder than anyone’s opinion — no matter whose. At the same time, there must be hurdles in place to stop decisions from being made without data backing them up.

It is also part of the culture to ensure that all teams understand the importance of experimentation and have the freedom to do their own testing. Remember the Booking.com example I mentioned earlier?

And naturally, another key part of the culture is communication. Like I’ve said before, the lessons learned from experiments are extremely valuable. Sometimes they are even more valuable than an increase in the conversion rate. How do the teams share experiences with each other? How do they discuss new experiment ideas? How do they prioritize? A solid culture will answer these types of questions.

Conclusion

I hope that, after this article, you now join in the opinion that Jeff Bezos and I share about experimentation and its relevance to any business.

This was an introductory piece about various aspects of the world of experimentation. If anything specific caught your attention or if you felt something was missing, drop me a comment!

Sign up for our newsletter for more posts like this delivered straight to your inbox: