Is that "web visitor" an agent or a human?
Unless you’ve been living under a rock, you’re aware that AI-Agents are likely coming to your website in the very near future. OpenAI’s Operator was the first “AI-Agent” to hit the market on January 23rd, 2025, which was made available to ChatGPT Pro users.
Google is working on Mariner and Microsoft is clipping at their heels with sights set high for agent activities embedded within Co-Pilot. All this leads us to speculate that we’re on the precipice of an AI-Agent assisted onslaught, which will be a new reality for businesses and consumers having these artificially intelligent bots do our bidding for us.
That led us on the Seer Analytics Team to perform some early testing to determine how marketers will be able to identify and track these AI-Agents using analytics when they inevitably show up on your websites.
Our early findings are interesting to say the least…read on if you’re curious.
Open AI "Operator" Visits were reported in GA4 as Bing/Organic
We tested OpenAI’s Operator on our Seerinteractive.com website, expecting GA4 to track it as direct traffic or (possibly) referral traffic from ChatGPT. Instead, it showed up as Bing/organic traffic.
That raised a bigger question—how can we accurately track ChatGPT's Operator Agent traffic?
What We Expected vs. What Actually Happened
Usually when web activity isn't human controlled, GA4 logs it as direct traffic (if no referral data was present) or ignores it as bot activity, which is usually filtered out of GA4 entirely. Instead, our test visits from Operator consistently appeared as Bing/organic, which was unexpected.
Yet, we’re learning that this is the nuance of trying to quantify agentic traffic. If you tell Operator to go visit a site like seerinteractive.com, our tests showed that it will reveal as direct traffic (with a source medium of direct / (none)). However, if you instruct Operator to go to Seer’s website, it is likely that it has to search for the site and it will use Bing to do so. Therefore it will change depending on where it decides to click. This could either be Bing/organic or we also learned that Operator will click on ads, so it could possibly be Bing/cpc.
We ran through a series of diagnostic checks—browser type, version, OS, user agent, and more—but none of them reliably distinguished Operator traffic from normal user sessions.
The Trap: To catch an agent we had to have an experiment set:
To understand how GA4 classified Operator traffic, we set up a controlled experiment.
We hypothesized that we would be able to identify AI-Agents by capturing a unique User Agent that would point us back to Operator. If we were right, our plan was to add a custom dimension for User_Agent within Google Tag Manager (GTM) and Google Analytics 4 (GA4). This, we thought, might help us distinguish AI-Agent website visits from human visits with a high degree of accuracy.
The Plan: How to track OpenAI Operator bot activity on our site:
✔ Selected Low-Traffic Pages – We planned to mimic a user session and chose pages with minimal daily visitors to isolate the test activity.
✔ Launch Operator Sessions – We have Operator visit the selected pages and click on CTA buttons that would trigger our key events under controlled conditions.
✔ Tracked GA4 Session IDs – We planned to capture both Client ID and Session ID through the GA4 payload found in the Network tab of the developer console to identify the Operator session (or sessions) during the test. Note: We did expect to see a series of sessions for each page that Operator visited, but thankfully this wasn’t the case!
✔ Analyzed Source/Medium – We planned to determine whether GA4 labeled the source of traffic as “direct,” “none,” or potentially ChatGPT/Operator referral.
The Headache: Here’s all the problems we ran into
- Operator uses Chromium – When we initiated the Operator session (from our Windows laptop), we immediately learned that it uses a Chromium browser. We did not expect this, so it threw a wrench into our planning.
- We Couldn’t Open the Developer Console in Chromium – We had planned to simply open up the dev console on the browser and identify the client ID and session ID within the payload tab, but this didn’t work for us. We know that Chromium does have a dev console, we just couldn’t get Operator to find it. So, we had to do a bit of sleuthing to identify the unique activity that gave this session away using the session ID in GA4 that was associated with our series of selected activities.
- User Agent Mismatch – To our surprise, we learned that Operator identifies as a Linux operating system using Chrome version 130.0.0.0.
- The Traffic Source / Medium is Not Consistent – These agents are very good at mimicking human behavior. If you tell them to go directly to a specific URL, they will do exactly that and that session will come through as direct traffic in Google Analytics. If you tell them to go to a company’s website and look at their product offerings, they would do what you or I would do. They are going to complete a search and since ChatGPT and Bing partner, they will use Bing to do this. Once they are on Bing they have the option of clicking an organic or a paid ad. Therefore, your traffic may come through as Bing / cpc or Bing / organic.
Key Finding: Some Operator AI-Agent shows up as Bing organic or paid traffic. If you see a steady increase in Bing traffic, it might not be humans visiting your site…It could be AI-Agents.
The solution: CHEQ had the fingerprint, and identified our Operator activity
So, the day after our test, we were anxiously awaiting the data to process and show up in GA4, when I received an email from our friends over at CHEQ.ai.
We’ve been in talks with CHEQ about how we were going to track AI-Agents and distinguish that traffic from humans and bots, but we both agreed that it’s still early days.
What I wasn’t expecting was that CHEQ sent me an email less than 24 hours after we ran our test with a complete session recall showing each of the pages we visited with Operator and the corresponding referrer path for each.
I was delighted to learn that the CHEQ tag on our website identified this single Operator session and they knew it in an instant. Naturally, I set up a meeting with Nick Gashi to learn more.
CHEQ has been quietly working on this in the background, but Nick informed me that our hacky little experiment was already something CHEQ's Threat and Traffic Intelligence Team was already detecting and classifying.
While we were running our isolated test on Seerintereactive.com, they were analyzing thousands of attributes about visitors, browsers, networks, and devices. Without revealing their secret sauce, we learned that they have developed a reliable fingerprinting method to identify ChatGPT Operator sessions, using a combination of attributes:
✔ Common Attributes – CHEQ’s detection engine observes thousands of attributes from a visitor’s browser, network, and device. They tested multiple Operator sessions across the million domains they monitor daily to identify patterns and commonality.
✔ User-Agent & OS Signals – Operator consistently identifies itself as a Linux device and currently runs on the same browser and version. By watching this and keeping up with changes, detection is possible.
✔ Cybersecurity Testing Patterns – CHEQ ran multiple proprietary security tests and found that Operator consistently fails specific security checks each session, providing a unique signature.
These combined signals allowed CHEQ to build a highly specific fingerprint for Operator traffic. More importantly, when backtesting against prior data, CHEQ found no sessions matching this pattern before January 2025, confirming that this fingerprint is unique to Operator.
Of course, this fingerprint may evolve, but the beauty of a solution like CHEQ is that they can catch evolving and new identifiers as they emerge -- this is the real power of their network effect (monitoring 1M domains + more than 6T signals daily).
Here’s what the CHEQ Chief Marketing Officer, Amy Holtzman had to say about their impending AI-Agent tracking solution…
What a time to be a digital leader. Engagement as we know it is a moving target with the rise of Operator, a wave of new agents soon entering the market, and the rapid adoption of all things AI. Accurately detecting Operator is just one example of how CHEQ remains committed to enabling companies to engage trusted audiences— human, agent, and machine—confidently and securely. - Amy Holtzman
Why This Matters for Analytics Teams
It’s certainly early days for AI-Agents, so now is the time to start figuring out how to track them, while the numbers are small, but we expect that over the next several months and throughout 2025, AI-Agents will become much more prevalent visitors on your websites and other digital properties.
These modern AI agents can navigate the web, mimic real users, and even take actions at scale, making it hard to tell if humans are visiting your site or if AI-Agents are – and who might be abusing it. It’s not unrealistic to imagine a not too distant future where Agents, not humans, will make up a lot of internet traffic.
Legacy techniques like CAPTCHAs, IP blocking, and user-agent filtering are largely ineffective against AI agents because these agents are engineered to look like real users. And, there will be Agents and AI traffic that you don’t want to block as LLMs index sites across the web to find answers and train their models. This makes tracking them with web analytics a difficult task.
As more AI tools interact with websites, analytics teams need to:
- Understand the Mix of Humans vs. Bots vs. AI Agents. AI agents will soon account for a growing share of website traffic, and traditional bot filters may not catch them. Marketers need to distinguish between human engagement, good AI agents (like search engine crawlers or personal assistants), and harmful bot traffic (like scrapers or fraud bots). Web analytics tools should segment traffic sources to provide a clearer view of real user behavior vs. automated interactions.
- Recognize and Adapt to AI-Driven Noise in Analytics. AI-Agents can distort key web metrics by mimicking human behavior, leading to inflated engagement rates, artificially low bounce rates, and misleading session duration data. Analysts should consider implementing AI-Agent detection and filtering within their analytics stack (e.g., bot management tools, log file analysis, and behavioral pattern tracking). New traffic classifications may be necessary to separate "human visits," "AI-assisted visits," and "agent-driven API calls" in reporting dashboards.
- Consider the Impact on Traditional Metrics and KPIs. Common KPIs like conversion rates, click-through rates, and attribution models may be skewed if AI-driven interactions are not accounted for correctly. AI-Agents can trigger events that appear as conversions (e.g., adding items to a cart, filling out forms) but may not represent real purchasing intent. Marketers should rethink their performance metrics—consider tracking AI-assisted conversions vs. human conversions to ensure accurate insights.
- Adapt SEO & Digital Marketing Strategies. Search and discovery are shifting toward AI-driven agents, meaning traditional search traffic metrics may decline as users rely more on AI-powered recommendations. Marketers must monitor how AI agents are accessing and interpreting content (e.g., optimizing for AI crawlers like GPTBot and OpenAI’s Operator in addition to traditional SEO). AI agents could become a new "referral source" in web analytics, requiring adjustments to tracking and attribution models.
- Evaluate New Approaches to Measurement and Data Governance. AI-Agent interactions require a different approach to tracking and auditing digital behavior. Businesses should establish transparent logging and AI-aware data governance policies to ensure ethical and compliant data collection. The future of analytics will likely involve hybrid models that combine human insights, AI-driven traffic segmentation, and real-time bot detection to maintain data integrity.
Key Takeaway: Marketers and analytics professionals must evolve their measurement strategies to differentiate between human engagement and AI interactions. This includes refining analytics methodologies, updating KPIs, and embracing new tracking mechanisms to ensure reliable insights in an AI-powered digital landscape.
What You Should Do Next
If you’re a marketer that’s concerned about how your business will identify and understand AI traffic to your digital assets, you’re not alone.
Here’s four things that you can do today to prepare:
- Identify AI-Referral traffic using custom channel groupings. Not to brag, but we at Seer have been encouraging you all to track AI traffic since 2023 when we first wrote about this topic. To do this, we recommend setting up an AI-search traffic channel in your analytics account. Whether you use Google Analytics, PiwikPro, or Adobe Analytics, we have you covered. By creating channel groupings and using our simple regex formula, you can be monitoring AI traffic to your websites today.
And as a bonus for GA4 users, once you create your channel groupings you can create your own Looker Studio dashboard by copying our template to Track AI Traffic. - Determine your own “When to Care” thresholds. It's early days...here at Seer, we're fond of giving practical advice that you can actually do something about. That's why we've developed "When to Care" thresholds. Our colleague Teresa wrote about this recently in her post Analyzing AI Overview Data w/ Paid Conversions: Finding When to Care, which explores how marketers can assess when AI-generated search experiences (like AI Overviews) and how they are meaningfully impacting businesses. We emphasize the need to focus on revenue-driven metrics rather than just visibility. So, a practical "When to Care" threshold for AI-Agent traffic might be when 5-10% of total traffic originates from AI agents or when AI-driven visits begin triggering key events on your websites and apps. If human traffic declines while AI Agents rise, or if conversion rates differ significantly between AI-Agents and Humans, it’s time to adjust your marketing strategies.
- Use a tool like Dark Visitors to understand AI-Agent traffic volume. While we’re still in the early days of AI-Agent detection, there are a small number of tools emerging that may prove very helpful. One of these tools that we’re testing is called Dark Visitors. They offer both a client-side and server-Side method of tracking AI-Agents and bots on your websites. By deploying their tag on your website, they are able to provide real-time analytics on AI agents, bots, and crawlers interacting with your website. One clever offering from Dark Visitors is the automatic generation of robots.txt files to manage bot access, protection against content scraping, detection of spoofed agents, and notifications for traffic spikes caused by bots.
As indicated in the screenshot above, they were even nice enough to send us an email notification when we tested Operator on our own website. While Dark Visitors is limited in its ability to tell you anything about the session activity of AI-Agents and bots, for marketers, this tool does offer valuable information revealing which AI agents and bots are accessing your websites, enabling you to determine “When to Care” and optimize strategies based on accurate traffic analysis.
- Expect that the most robust solutions will come from Bot detection vendors like CHEQ. Detecting AI-Agents and discerning what they actually do when they arrive at your digital properties will continue to be a challenging endeavor even for the most savvy Analysts. We expect that experts in the field of cyber security and bot detection will come up with the most scalable and sustainable solutions. Watch for developments from companies like CHEQ, HUMAN, DataDome, Kasada, Imperva who are on the forefront of bot detection and fraud detection.
- Full Disclosure: Seer and CHEQ are partners and we work with them and refer our clients to them because unlike most of the vendors mentioned above, they work with marketers – and security teams. CHEQ understands your marketing objectives and how the impending onslaught of AI-Agents and GenAI scrapers will impact your ability to effectively market to humans. Additionally, CHEQ has an integration with Google Analytics and Adobe to not only identify agentic traffic, but to also enable you to determine what pages the AI-Agents visit, how they engage with your digital properties, and when they convert. This provides outstanding insight into what you probably won’t see with other tools.
So as you embark on discerning the humans from the AI-Agents, we’ll continue tracking how analytics solutions handle AI-driven traffic and report back on any additional insights. If you want to follow along with our findings as they’re published, subscribe to our newsletter for all the latest and greatest thinking from the Seer Team.
If you’re seeing similar issues, let us know—this could be part of a larger tracking challenge in digital analytics. We want to hear from you!
