How Will AI Generated Content Impact the ROI of SEO & Content?

If you are tasked with creating a POV around the risk/upside to SEO, content marketing, or paid search in a ChatGPT world, then this post is for you.

This is not a post about:

• Ethics
• Disclosures
• Microsoft’s investments
• Models
• AI content vs human content

This post is focused on economic impacts, ROI impacts, and risks to your company

I wanted to give you approaches and ways of thinking/tracking/sleuthing to aid in determining the potential impact rooted in existing data you already have as a search professional, but probably aren't using to answer the questions you are suddenly getting.

I’m going to try to take that approach to ChatGPT – specifically through the lens of how it impacts the ROI of SEO and investments in content marketing. 

The questions I am trying to answer for myself and my clients are:

• Should I be worried that the value of my existing organic search rankings will be diminished?
• Short term – Are my competitors using AI in content workflows and beating me today? (Impact the ROI equation of ranking)
• Long term – If Bing or Google move to a world of AI answers on the SERP vs blue links (Impacts the value for the entire market)
• How can I determine which content is worth producing and ranking? As a result, what content should I let AI answer (either by me or by a competitor)?
• What types of keywords are most likely to be easily answered by ChatGPT/AI – and how much of a hit might that be? 

Short-term Risk in SEO and Content Investments

Should I be worried that the value of my existing organic search rankings will be diminished if competitors start generating content using AI?

Yes. You should have been worried about the downtrend in ROI of SEO well before ChatGPT, as it is well documented that Google has placed more ads, more answers, and more rich snippets around organic rankings, resulting in “zero-click” searches. I cannot imagine a world where all these distractions that show up around the SERP result in more clicks on organic listings.

Now, let’s talk about ChatGPT or the use of AI-assisted content. The value of using SEO as a way to rank high on Google to capture traffic will likely be diminished for many of the sites where people don’t need to search anymore and answers are surfaced through ChatGPT. The question is: to what degree?

How to identify the SERPs likely to be disrupted?

Step 1: Find organizations leveraging AI to generate content

Here are a couple of widely covered examples in the banking sector, which is where I’ll focus since we have clients that are banks and we have worked with large publishers in the financial sector.

The disclosure helps you to find them at scale:

There are a lot of ways to try to catch AI-generated text. (Here is a great roundup from Chris Cemper.)

This thread by Rich Tatum of Edgy Labs outlines how he wrote a post using ChatGPT and then ran it through several tools to determine how well they could catch the AI. Tools like these are going to be critical to building a set of competitors to "watch" over time.

However you find URLs, find them, and begin to monitor the SERP for a bespoke analysis vs. a generic one. (Personally, I think it is worth the work).

Let’s step our game up a bit and scale this - because running “site colon” searches is time-consuming.

When we talk about risk to the value of SEO investments – we must take the AI-generated content we identified above & compare those pages to our existing rankings to see: 

• Which competitors are able to outrank us with AI-generated content and 
• How often are they able to outrank our existing content

Let’s use SEMrush’s tools to dissect how to determine AI “assisted” content performance.

Step 2: Join Your Data

To me, when talking to execs about risk, dollars, and cents make the most sense. 

Typically an SEO working with siloed SEO data might see these AI-assisted content pieces and think, “Oh my god! The AI content built by our competitors is outranking us for this list of keywords with X search volume.” I’ll show you below how joining paid data could help you avoid that mistake in your POVs.

My preference is to join the PPC and SEO data together to determine risk. Why? PPC data gives me these 2 things to value content that search volume does not:

• Propensity to convert (These words drive X leads / MQLs)
• Value in the market (last year you saw enough value in these words to spend X dollars)

Despite AI-generated content outranking us, maybe these keywords don’t drive conversions.  This could change the way I position the strategy. 

That approach doesn't scale, so lets scale it:

That approach is decent, but across several clients and 1000’s of URLs, I will miss as many as I catch. I want to catch all of them at scale, while also alerting my team and providing a consistent level of service to clients.

That’s where our data warehousing, and insights platform, comes into play, I was able to go from hype to hypothesis to client impact in under 2 hours, you could do this easily in Power Bi across clients too, but I'm showing what we did in Looker.

Wanna get real fancy? Consider these options in your risk assessment:

Maybe queries best answered by AI don’t get clicks past position 2 or 3 anyway.

• Adding in Google search console data would give you a sense of click-through rates too. Maybe the types of answers AI is right for are the types of keywords people don’t dig that deep on anyway (low CTR). This could mean that ranking below the Top 2 answers isn’t worth it. You could do this by joining search terms from GSC to search terms in your competitor ranking report. You could also use answer boxes as a proxy for this using historical data, what does your CTR look like when answer boxes show up vs when they don’t?
• Adding in analytics data, you could potentially see that you got MORE traffic after AI outranked you because AI might be generating an insufficient answer. Finding the competitors cranking out low-quality AI could result in more engagement for you. It is one way to not freak out because you are outranked on Google, but instead to see the effect on your traffic when it happens.

Long-term Risk in SEO and Content Investments

The other side of the financial coin, is what if this upends search altogether and people stop using search and start using ChatGPT? First, keep this in mind from JPMorgan’s analyst, we don't know the cost of ChatGPT yet, and it is more expensive than Google's infrastructure to answer 1 query:

Ok, with that said, how would I make a case to help an exec understand the long-term risk to the value of ranking?

What makes a query disruptable/easy to answer completely with AI?

Above, we just showed you how to manually look at a SERP looking for clues. But there are other ways to do this at scale.

AI doesn't have hypotheses, YOU do.

Wanna protect your job in content or SEO? Start having more hypotheses and finding ways to validate or invalidate them.  Let me give you 2 (simple and bad) examples of hypotheses, I played around with that could impact the ROI of paid and organic search.

I believe People Also Ask in the Top 2 positions on Google is a flag, or an answer box is an indicator of content that is easily disrupted by AI content - I can tell my clients what % of their ranking search terms trigger those “warning signals.” Then, combining it with paid data can help me to advise them on how many of their conversions could be at risk, and if conversions or leads are at risk, revenue goals may be missed and investments/strategies may need to shift.

Hypothesis 1: Let’s say I believed that PAA’s in Position 2 meant that those queries were somewhat more disruptable than instances where PAA is lower or not on the page at all. Now, I can see the percentage of client spend/conversions that matches my hypothesis around PAAs.

I’m now able to go from client to client instantly.

For one client asking my opinion about AI assisted content & risks I might say:

"Hey, if you believe that my hypothesis is a good proxy for query disruption, you are in the top 5% of ALL my clients for whom conversions are likely to be impacted. This means we should have a strategy session in the next 3 months."

For the other asking my opinion about AI assisted content & risks I might say:

"Hey, 66% of your keywords are affected by this hypothesis, but it only represents 17% of your conversions. This puts you in the bottom 10% of my clients. As such, I will monitor this every month and if the percentage of conversions impacted exceeds 25%, then we should set up a strategy session in the next 3-6 months."

Using the same data set, updated monthly allows me to have very different conversations. This is for another day, but this is one of THE value of an agency, the ability to have access to data quickly across 100+ clients enables me to see a range of who should have a conversation today due to near term risk and who doesn't need to worry just yet. 

The other thing I love about our platform is if one team member has a killer hypothesis, we can shift priorities and build the dash for every client at scale without our team members having to learn how to build the dash.  (Because they’ve got too many other things to do!)

Hypothesis 2: Let’s say I believed that Answer Boxes ranking in a certain position were the key to uncovering easily disruptable rankings. I could do the following:

In the dashboard above, I am able to articulate to two very different strategies to two clients both hovering at a 20% of queries that are “disruptable.” Client 1 has a  20% rate of disruptable queries but only 1% of conversions. They are at less of a threat than Client 2, who also has 20% of disruptable queries – but at 6% of their conversions. 

Note: I gotta learn how to make quadrant charts in Looker, cause I would love to have 4 quadrants. 

What types of keywords are most likely to be easily answered by ChatGPT/AI?

I like to look for similarities in the SERP. Find a page or two or three and make a prediction. Do you think it will last, do you think it is low quality, etc.? There we are again, have an opinion or a hypothesis.

If you are looking for something a bit more advanced:  

1. Use SEOquake after you run your site: search and export all the URLs that matched your AI patterns.
2. Save them as a CSV
3. Take your client and download all the rankings where the AI-assisted content ranks

Let me know if you would like me to record a video on how to do this in SEMrush + PBI.

I would take the keyword rankings of these pages (pulled from Ahrefs or SEMrush) and Theme them using N-grams, here is a great tool. Use this information to see what percentage of them have words like:

• pros
• cons
• what is
• how does
• etc.

As you can see in the Ahrefs screengrab below, the AI-generated content performed best on words like those above for this 1 piece of content. You’d then have to go through the rest looking for similar patterns.

Additionally, you can always dump all the keywords into a tool to N-Gram them looking for commonalities.

Ok, so whew, that is all I have for now! Let’s get after it! 

How are you discussing ROI, risk, impact? 

My next post (when I can come up for air) will likely be about economic models of human content vs AI content and the considerations you should have going into making that decision. If you only need to make (or have budget to make) 20 pieces of content this year, then maybe all this AI stuff is overblown. (As long as you aren’t making 20 pieces of content easily answered by AI). However, if you see the need to make 100 or 200, then maybe you do need more assistance to scale the work of the team…But I’m getting ahead of myself there! See you in the next post!