Dima Durah
Oct 27, 2024
How We Accidentally Built a Better SEO Tool: Agentic Web Report
The way people search for information online is evolving. AI-powered search agents like SearchGPT, Perplexity, and Gemini are rapidly gaining popularity, with recent reports showing that AI searches have taken up a significant share of Google’s traffic. This shift is leaving many businesses scrambling to figure out how to maintain their visibility in an AI-driven world.
This is where our "story time" begins. We didn’t set out to build a Search Engine Optimization tool, but we accidentally did—and within days, we had our first paying customer.
Our team isn’t even equipped with traditional SEO expertise, but that actually worked to our advantage. Here’s what happened:
The first step was taking a hard look at what wasn’t working with the current state of search. Traditional SEO strategies are about getting your content ranked highly on search engines like Google by optimizing for keywords, links, and metadata. While this has worked for years, AI search agents are disrupting that model.
Instead of relying solely on algorithms, AI search agents leverage large language models (LLMs) that look for keywords, interpret and respond to natural language questions. In short, users are no longer just typing keywords; they’re asking full, conversational questions.
So, we asked ourselves: how can we optimize for a model that understands context, associations, and meaning beyond words? And more importantly, how can we ensure our partner brands stay visible in the relevant LLM search results?
To answer these questions, we reached out to some startup CMOs. We didn’t have a detailed plan—we just started talking and building on the fly.
In a matter of hours, we created the first iteration of the Agentic Web Report, a Search AGENT Optimization (SAO) tool.
Here’s how we approached it:
Identifying Key Questions: We started by thinking about the kinds of natural-language questions potential customers would ask AI search. Unlike traditional keyword-based SEO, this required us to think in terms of conversation—what problems are brands solving for customers? What specific questions might their customers ask when looking for their solutions?
Formulating AI-Specific Queries: Once we identified key questions, we used AI to translate them into queries that large language models (LLMs) would understand. This meant creating thousands of human-like, topically specific questions that align with how people naturally interact with AI search agents.
Querying Multiple Search Engines: To get a broad understanding of how our customer's content was performing, we ran these queries across various search engines—Google, Baidu, Naver, Bing, and DuckDuckGo. This allowed us to see how our partner brand’s content was showing up in different markets and contexts.
Analyzing Web Results: The next step was analyzing the results. We uncovered related questions, top sources, and key mentions that LLMs were surfacing in response to our queries. This gave us valuable insights into how AI search associates our partner’s brand with relevant topics.
AI Discoverability: Using the data we collected, we made strategic recommendations on how to tweak existing content to target both the traditional web (where search engines rank based on algorithms) and the Agentic Web—the space where LLMs make unexpected associations. For example, we found that an allergy medicine might be linked to something as unrelated as a mattress brand based on user behavior and LLM’s learned associations.
Testing with LLMs: We then sent our newly optimized content back into the LLMs to see how they responded to our revised strategy. This feedback loop allowed us to continuously fine-tune our approach.
Analyzing AI-Driven Insights: Finally, we dug deeper into the LLM insights we received. These results revealed new patterns, user behaviors, and associations our brand partner hadn’t previously considered.
Early Results:
It’s still early days to talk about the long-term impact, but the initial signs are promising. We’re already seeing an uptick in traffic. But what we’re really excited about is the deeper understanding of how LLMs interact with users. This means our Living Assets are going to be effective for both the traditional web as well as the Agentic web—they’re basically beacons that AI search can latch onto, delivering a curated brand experience within a GPT chat context window. In other words, our brand partners could curate their brand experience directly in an LLM chat (Claude, ChatGPT, Gemini).
What’s Next?
We’re launching our first fleet of Living Assets with SMBs that are tired of creating Lead Gen content that gets them nowhere. If you’d like to run the same experiment, we’ve created a free standalone tool for you to use. Reach out!