How to Bring CRO / Experimentation In-House

What is Conversion Rate Optimization?

Our simplified definition is "the disciplined and science-driven practice of increasing the percentage of users who perform a desired action on a website or app." It's a continuous process of:

1. Gathering data that helps you understand how to improve your website/app/campaign/etc conversion rate
2. Making the changes to hopefully fix the problems / opportunities you've identified
3. AB testing those changes to understand if they really worked
4. Repeat!

CRO Agency or In-House Team

Recently, there's been a trend of trying to do CRO in-house instead of paying an agency to do it. The goal of this article is not to tell you if you should hire an agency or bring everything in-house. We'll talk about this some other day.

💡 Thinking about building an internal CRO team? This guide is for you!

I'll show you what you need to build a successful experimentation program to boost innovation and revenue at your business.

Who Could Benefit from In-House CRO

The majority of business models can successfully do CRO either internally or with an agency, as long as they do it right / the agency is good.

But for some businesses, it can be more challenging to use an agency.

Regulated industries

If, for example, you are in a heavily-regulated industry that limits your capacity to share data with an external partner, you might end up making your agency's task almost impossible.

Healthcare and big financial institutions commonly face this scenario. Even though it's still possible to share valuable data with partners to develop a good CRO program in these industries while still being 100% "compliant," it's especially hard to get approval to do this at some companies.

High-traffic sites

Another scenario where you might want to do things in-house is when you have massive traffic (millions of monthly users) to test on. This gives you the potential to run a number of tests that most agencies won't have the necessary team to explore (yes, it's hard for agencies to find CRO talent too).

It doesn't mean you can't partner with an agency to help you run as many experiments as possible. But you'll probably want internal teams running tests too so you can accelerate your growth and learning speed.

How to Build an Experimentation Program

Keep in mind, you don't start an experimentation program and magically begin running 100,000+ tests per year on every area of your business.

There's a long, expensive, and tricky road to get there. To be the next Google one day, we have to start somewhere, right?

Today, I'll focus on showing you how to start doing CRO / Experimentation internally. But you must understand that it will only be the first win. At the end of the article, I'll discuss how to keep evolving your program.

💡 These are the 4 topics to address to improve your chances of success with Experimentation in both the short and long term:

1. Organizational Culture
2. A Cross-Functional Team
3. Reliable Processes
4. Tech and Data Stacks

1. Organizational Culture

For some companies, this might be the hardest part. But it's impossible to have a good experimentation program without the right culture.

Be data-driven

If the decision-makers refuse to be data-driven, why would you test stuff anyway?

And to be really data-driven, it's not enough to just stick to the ad copy that performs better on Google Ads. It's much deeper than that.

It's about being open to running experiments that will guide decisions at every level of the organization where it would make sense. Including at the strategic level.

By the way, if decision-makers are not humble enough to accept when data proves them wrong, an experimentation program won't go far. I've seen this more than once. Even from leaders that speak at conferences about "how to be data-driven.” Ego is tough to handle.

Innovate at every level

Decision-makers also have to be humble enough to democratize innovation.

Letting people around the organization run experiments to guide decisions on every level of the business means "losing control” over a lot of the innovation that happens at the company.

At the best companies, this is actually excellent. And I'd argue that this is the only way to innovate fast enough to stay relevant at some highly competitive industries. But some leaders are not willing to accept this.

Be customer-focused

The right culture is also about having a crazy focus on the customer. Because, in the end, optimization is about learning what customers like more or what makes things easier for them.

There's no greater platform for pleasing your customers than experimentation. As Expedia's CEO puts it:

“At any one time, we’re running hundreds, if not thousands, of concurrent experiments, involving millions of visitors. Because of this, we don’t have to guess what customers want; we have the ability to run the most massive ‘customer surveys’ that exist, again and again, to have them tell us what they want.”

Sometimes, customer feedback won't be the best news for business in the short term, maybe reducing profitability or forcing teams to pivot strategies. However, the best experimenters can understand that by being customer-centric, they tend to win in the long run.

Have leadership buy-in

It's critical for an Experimentation program to have 100% buy-in from leadership.

You'll need C-Levels by your side to:

• Be able to influence the processes and people that have to adjust their behavior for experimentation to thrive. If you're Marketing, but all your product teams refuse to AB test their changes, you're not going to affect the bottom line as much as you could.
• Get the funding you need for the program. As you'll see below, there are many costs. And they aren't negligible.
• Create a culture that values and incentivizes learning as much as test wins. Without active cultural, financial, and career motivation, it's hard to get team members to genuinely pursue innovation like at Amazon, Google, and the other great experimenters.

But I know culture can be subjective. And sometimes, when not sure how to behave, benchmarks can help as role models.

Look at Booking.com, for example, known for having a great experimentation culture and running more than 25,000 tests per year.

There, any team member can run tests without management's permission. Actually, some great wins that generated millions of dollars in revenue came from test ideas that managers didn't believe in (but let them happen anyway).

David Vismans, Chief Product Officer at Booking.com, summarizes what Experimentation expects from leadership:

“If I have any advice for CEOs, it’s this: Large-scale testing is not a technical thing; it’s a cultural thing that you need to fully embrace. You need to ask yourself two big questions: How willing are you to be confronted every day by how wrong you are? And how much autonomy are you willing to give to the people who work for you? And if the answer is that you don’t like to be proven wrong and don’t want employees to decide the future of your products, it’s not going to work. You will never reap the full benefits of experimentation.”

2. A Cross-Functional Team

Now let's get more practical and talk about your CRO/Experimentation team.

If you want to build a CRO team, you'll need at the least the following skills well-covered by its members:

• Copywriting
• Design
• User Experience
• Digital Analytics (tracking and analysis)
• User Research
• Statistics
• Front-end design (HTML, CSS & JS)
• Communication
• Project Management

Considering the professionals available in the market today and the average volume of work, I'd recommend having at least 2-3 people covering these skills to get started. I know it's a lot but trust me: miss any of these, and your team will certainly face trouble.

Of course, details could change from company to company. But the structure could look like this:

The cost for this basic structure also can vary a lot. Not only because of the experience level you'll be aiming for but also because you might not start with everyone full-time (even though you should). But I wouldn't expect to spend less than $200-$250K on salaries (not including benefits and corporate overheads) for this structure. Less than that could mean a team with not enough experience to give the program a good chance to win.

And it's important to remember that this is the recommended team structure to get started with CRO / Experimentation. More mature companies tend to use less centralized approaches to enable all teams to run experiments. We'll talk more about it later.

And on many cases, this team won't have enough bandwidth to run all the tests a company could be running considering their traffic volume. But again: it's the first step.

3. Reliable Processes

With the right team available, it's time to make sure they'll work in the smartest way.

This is also another topic we could talk about forever. But for the scope of this article, let's just say that your team's process will have to cover at least these activities:

Research

Where will the data that backs up test ideas come from? What's the process of gathering insights from that data? How will you run qualitative research methods like usability testing or email surveys?

Good research is a key activity for any decent CRO Program. Make sure you have an efficient and clear process to generate insights. If you fail here (AKA, have terrible test ideas), everything that follows will fail too.

Define KPIs and guardrail metrics

The best programs have KPIs aligned with the company's priority at the time. Always use a relevant KPI and that everything on the program gravitates around that metric.

You also have to define and check guardrail metrics. These are important metrics for the business that might not be your KPI but should be tracked on every experiment to make sure you are not improving something while negatively influencing something else. Example guardrail metrics could be related to Lifetime Value, User satisfaction or site performance.

Building hypotheses

The team must clearly define what an acceptable test hypothesis is, what information has to be there, and where to store it.

Prioritizing

There will always be more test ideas than what you can implement. That's why prioritization is key. Starting with standard frameworks like ICE is acceptable. But ideally, you should have a framework customized for your specific business needs.

Implementing the change

Make sure that the time it will take from when you decide what the next time is going to be from the time it's launched is as short as possible. This is crucial to reach a good speed on an experimentation program.

The process must ensure that designers and devs are ready to kick off when the idea is decided and that you have a killer QA process to avoid launching tests with some problem that will force a restart.

When to stop a test

Define clear rules that have to be followed for someone to stop a test (declaring a winner/loser). Skip that, and you'll probably have trouble with cherry-picking and other dangerous mistakes.

How to analyze a test result

AB testing tools reports can be very dangerous if the person analyzing them doesn't really understand the statistical nuances there. Make sure you have some kind of checklist or quality control initiatives to avoid the risk of teams making decisions on “false” data.

Sharing learnings

Sometimes, the learnings from tests can be more valuable than the actual increment in conversion rates they generate. For example, if mentioning a new benefit on a Landing Page worked well, why not test that same thing on your Ad copy?

Experimentation teaches you a lot about your customer. And if you really want your company to be customer-centric (you should), there has to be a process to ensure these learnings will reach all the other teams.

Documenting

Another process that helps spread learnings is documenting. This also helps a lot when onboarding new people. Can you think of a better crash course for new hires than a list with the approaches we already know that work (or don't) with your customers?

4. Tech and Data Stacks

As you know, data and technology are the backbones of an Experimentation / CRO program. Some critical activities on that pillar are:

Deciding if you should build or buy an AB testing tool

The right choice will depend on your context. Depending on the number of specific needs or the severity of your compliance rules, you might need to build your own AB testing tool.

But in most cases, it's totally OK to start with one of the many good AB Testing tools out there. And as your maturity grows, you'll have a better understanding of what your specific needs really are, and then you'll be able to build your own solution, if it makes sense. When buying, just make sure to choose the tool that better integrates with your current stack.

Integrating with customer data

For some business models, it's very limiting to run tests using only "web" metrics (i.e., signups, clicks, free trials.) You'll make much better decisions if you also understand the impact of changes on customer metrics like revenue, LTV, and satisfaction.

To do that, you should properly integrate your AB testing tool with your CDP or wherever your customer data lives.

Allowing tests on any area of the conversion funnel

Experimentation should happen in every area of the business. Limiting tests only to landing pages or signup forms is a terrible idea. It's like having Cristiano Ronaldo on your team and telling him he can't shoot (yes, I’m Brazilian and always have a soccer metaphor on hand).

Make sure your teams have the ability to scientifically optimize every possible step of the customer journey. Checkouts, product, customer service platforms… Everything should be optimizable.

Tracking everything needed

Without proper tracking, you don't have data. Without data, you can't run a serious Experimentation program.

Track everything relevant on the customer journey, never publish something without tracking, and regularly check if everything is still working.

Estimating Costs and ROI

You might be asking: “OK. That sounds like a lot. How much will that cost me?”

💡 To estimate your yearly costs, you have to consider:

• Your CRO team costs (US$200-250K) as mentioned above.
• Other team members' hours that will be needed to support the process you come up with (PMs, devs, BI, leadership…).
• Total cost of your tools and tech needs (for AB testing, Research, Data, and so on).

I know it sounds like a lot. And it can be. And that's why you have to get Experimentation right.

If you want a simple way to estimate that return, you can play around with what you believe your team can accomplish using this formula:

[ Experiments Ran per Year x ( Percentage of Experiments Won x Avg Additional Revenue per Experiment Won Annually ) ] - Total Yearly Cost for the Program

As the formula shows, your ROI will depend heavily on:

• How fast you can move (the number of experiments). That's why you need a great process.
• How good your experiments are (percentage of wins and average financial impact). That's why you need great people, with access to data, in a culture that lets them innovate.

Don't Stop Experimenting

The Experimentation Flywheel from Microsoft, Booking, and Outreach.io explains this as a cyclical and evolving process that includes:

1. Run experiments to support decisions
2. Measure results/value
3. Evangelize Experimentation
4. Invest in Experimentation infrastructure
5. Decrease cost of testing new areas

As the name suggests, once an Expermentation team goes through the fifth step they start back at one.

Image source: It takes a Flywheel to Fly

They make it clear that high experimentation maturity is a transformation that is so big for most companies that it needs to happen iteratively. That's why I'm highlighting that it's really important to start things the right way. It will take time (and wins) to reach the end goal.

Note that as your maturity grows, you'll need to reinvent your process and team structure. First, make sure you get the foundations right with the tips from this article.

Conclusion

As mentioned earlier, all you've read in this article covers a basic setup to kick off an internal CRO / Experimentation program.

Once you start, you have to keep evolving to higher levels of maturity to become the next Google/Meta/Amazon/Netflix/Booking/Spotify.

Additional Resources

 For more tips and tricks on CRO / Experimentation, keep reading:

• How to Become (or Train) a CRO Specialist
• What’s All The Hype About Experimentation?
• The Untapped Potential of Experimentation
• What to Do Without Enough Traffic for A/B Testing

Looking for an Agency Partner?

If your company is interested in creating a successful in-house program, Seer’s CRO team can get you from zero to functional in four months.

Contact us to discuss how we can work together!

^ back to Top

Sign up for our newsletter for more posts like this in your inbox: