SEMrush just dropped a massive study analyzing 80 million ChatGPT queries, and one stat caught our attention: 46% of queries triggered SearchGPT.
Our (much smaller) research indicated that number was closer to 30%.
Why it matters:
For those wishing to optimize for generative engines, the higher the better! SearchGPT offers the best entry point to influencing generative engine results.
SEMrush’s data suggests there’s even more potential for generative engine optimization (GEO) than we previously believed.
The following research suggests that we’ve got some clear correlations present with how exactly brands get mentioned in those SearchGPT citations.
With OpenAI’s partnership with Microsoft, there’s been plenty of speculation about how much Bing’s search results influence the citations you see in ChatGPT’s answers. So, we decided to dig deeper.
By analyzing over 500 citations, we uncovered just how closely SearchGPT’s citations align with Bing’s top search results—and what that means for your SEO strategy moving forward.
Why ChatGPT’s Citations Matter
In a previous post, we walked through how ChatGPT evaluates questions to determine if a web search is necessary to answer the question and our framework around those factors.
Using this framework, you can start thinking about what strategies to focus on for different types of questions - is ChatGPT more likely to use training data or SearchGPT to answer this question? If ChatGPT’s web search comes into play, you need to start paying attention to Bing.
[TIP] Use the FLIP framework to understand which strategies you should use to optimize for answers from training data or answers from web search.
Bing isn’t always the first search engine marketers think of, but with OpenAI and Microsoft’s partnership, it’s playing a much bigger role in how SearchGPT answers questions. If you’re not tracking Bing’s top results, you could be missing out on visibility in AI-generated responses.
What We Wanted to Find Out
When ChatGPT performs a web search and uses citations to answer questions, where exactly are those citations coming from?
Yes, OpenAI has a partnership with Microsoft, but what query does it search for? How many SERP results does it review? What types of websites? What types of SERP features does it look at?
How We Did It
1. Asked questions to ChatGPT
We used the same data from our study evaluating how often ChatGPT uses its training data compared to searching the web, where we scraped the answer and all citations for each question where a web search was triggered. We would have loved to use our Generative Engine Tracker for this project, but SearchGPT citations aren’t yet available through OpenAI’s APIs as of the time of this publishing.
2. Tracked the same questions as queries in Google and Bing
Then, we took the same exact questions and tracked them in Google and Bing, capturing all SERP results for each question. We were able to do this more easily using SeerSignals.
3. Created a cross-channel LLM and SEO dataset
Finally, we joined our ChatGPT data to SERP data from Google and Bing - we weren’t looking for correlations here, we were searching for exact matches, joining the data on question to query and citation to link for 500+ citations.
Most SearchGPT Citations Match Bing’s Top 20 Results
87%+ of SearchGPT’s citations matched Bing’s top organic results when the same exact question was used as a query, with most of those results appearing in the top 10 positions. We did see many citations ranking in positions 11-20, so don’t count out page 2 yet!
Google only saw a 56% match with a median rank of 17 and average rank of 28 for results that matched citations. More on that below.
No surprise here, given the OpenAI-Bing partnership. But this research adds context: it’s not just Bing, it’s Bing’s top rankings driving what SearchGPT cites.
For a platform that’s dominated search for decades, Google’s 56% citation match in SearchGPT may be a wake-up call.
So why not a 100% match?
- Query simplification: we had a very high exact match rate by simply sending the same exact questions we ran in ChatGPT to Bing. However, I believe that for longer or more complex questions, ChatGPT likely simplifies the query. Posing a question to ChatGPT like “what are the best auto loan lenders for those with bad credit?” often appears to simplify a query like “best auto loan lenders bad credit” before searching Bing.
- Data cleanliness: although we included cleaning steps to improve joining citations to URLs (like lowercase, trim, removing trailing slashes, removing URL parameters) we may have seen a handful of mismatches from data cleanliness.
- Unknown unknowns: there are likely other factors that we could dig into with future testing. For example, we identified several Wikipedia citations that didn’t match to Bing SERPs - we could hypothesize that ChatGPT includes all citations used in an answer, including combinations of web and training data citations.
All Cited Sources Were Present in Organic Search Results
Almost all of the citations were tied to an organic result in Bing. Although we extracted the top-ranking results for our analysis and found a handful of matches on PAA, News, and Organic results - all of these SERP Features were generated from an existing organic page.
Most Cited Websites Were Affiliates, News, and Aggregator Sites
The majority of our questions in this dataset included modifiers like “top” and “best”, so it’s not surprising that the top-ranking Bing results skewed towards the types of sites who often publish roundup or listicles for comparisons.
Even if our data was skewed towards these types of sites, it still proves that affiliates & aggregators have value in the ecosystem of the Internet - LLMs can’t replace affiliates and aggregators if they’re actively using them.
What Questions Do We Still Have?
What Domains Were Ranking on Google but Not Bing (and Thus, Not SearchGPT)?
Remember this is a limited sample of 100 queries, so our insights on this topic are directional.
That said, when we looked at the webpages and domains Google most strongly aligned with the query (the top 5 organic results) and compared those domains to those present in Bing, we found some interesting omissions.
What Domains Were Cited by ChatGPT but Not Ranking on Bing?
We identified several citations for Wikipedia, aggregators, and news sites that were cited as sources by ChatGPT, but were not ranking on Bing when we searched the same question.
We’re already in-progress expanding this research across 10K queries to answer these questions and learn more about the types of results that ChatGPT uses to answer questions when it searches the web!
As we expand this research, we’ll be uncovering even more about how ChatGPT pulls citations and what that means for your SEO strategy. Want first access to our latest findings, strategies, and insights?
Sign up for our newsletter and be the first hear our testing results.
What This Means For Your LLM Optimization Strategy
If your content strategy is still all about Google, it’s time to diversify.
SearchGPT’s reliance on Bing is a wake-up call for anyone optimizing for visibility in AI answers.
Here’s where you can start:
- Use the FLIP framework: Understand what types of questions are likely to trigger a web search in ChatGPT by using the FLIP framework or testing them directly in ChatGPT. You’ll likely use a different approach to increase mentions when ChatGPT uses only its training data than you would when ChatGPT searches the web.
- Track Top Questions in Bing: With SearchGPT pulling citations from Bing’s top organic rankings, keeping an eye on the SERPs should be part of your strategy.
- Simplify for AI: Remember, tools like SearchGPT might simplify queries. A long question like “What’s the best auto loan lender for bad credit?” might become “best auto loan lender bad credit.” Track variations in Bing to understand how results differ.
- Don’t Only Optimize Your Site: When it comes to SearchGPT, it’s not enough to focus solely on getting your own site into Bing’s top results. The reality is that SearchGPT’s citations pull from a mix of sources: affiliates, aggregators, PR/news sites, and more: If your brand isn’t already ranking in the top spots, work with the sites that are. Partner with aggregators and leverage your PR team to target news outlets and affiliates that are trusted in your industry.
Want to learn more about how AI impacts search visibility?
Join us on 2/26 for a live webinar where we’ll share deeper insights from our AI research, register here.
