Competitive analysis is tedious. You pull your competitors' ranking keywords, backlink profiles and top pages, analyze them, write it up, and forget about it two weeks later.

The pulling and sorting is exactly what agents are good at, so that's what I hand them. I run competitor analysis on demand, when a project needs it, with agents (Codex and Claude) doing the API pulls, deduping, filtering and first-pass clustering under my direction. The pull turns out to be the easy part, and the raw output is mostly noise.

The pipeline

Step 1: Define competitors, then check that definition

Start with the competitors the client names, then add the domains that actually show up in the search results for the terms that matter. Those two lists are rarely the same.

Step 2: Pull their data by API

Step 3: Dedupe and filter

Most of the work happens here (more on that below).

Step 4: Cluster and write it up

The filtered set gets clustered into topics and turned into a short memo: the real competitors, the gaps worth chasing, and a prioritized list. I review it and decide what goes into the roadmap.

What the raw data actually looks like

For a commercial real-estate software client, the agents pulled organic keyword sets for 15 competitor domains plus three domain-gap batches. That produced 2,120 raw competitor keyword rows and 1,692 unique keywords. After filtering for relevance, 473 were left, and the gap list came down to 61 unique terms worth considering. The pull also surfaced competitors the client hadn't named, which changed who we were actually benchmarking against.

For a South Florida estate-care startup, the founder sent a list of competitor sites. The first wide keyword harvest used competitors discovered from the search results, and only some of his named domains showed up in it naturally. So we ran an explicit gap against all 12 named domains and crawled 88 of their pages. That pass returned 8,400 keyword rows, and a lot of it was real estate listings, job searches, streaming and out-of-market terms, none of which you'd want to plan a site around. After cleanup, plus deliberate service-and-location expansion, the planning list was 1,330 keywords.

The lesson from both: if you pipe a raw competitor export straight into a model and ask for "opportunities," you'll get confident recommendations built on noise.

What works well

Agents are great at the volume work. Pulling a dozen domains, deduping thousands of rows and grouping them into clusters is hours of spreadsheet work done quickly, and it's consistent every time.

Agents are also good at spotting structural patterns once the data is clean. In an audit for a real estate brokerage client, the read was that local competitors often had weaker link authority than the client but won specific searches with more specific pages. That points to building the right pages rather than only chasing links.

And agents don't get bored. If you want the same analysis re-run next quarter, it's the same script.

What doesn't work

Agents misread what an opportunity is. They'll flag a keyword because a competitor ranks for it, even when there's no meaningful search volume or it doesn't fit the business. Filtering and review are not optional.

Agents don't understand business context. A model can recommend a keyword your actual customer never searches, and catching that takes someone who knows the business.

And everything depends on data quality. If the keyword tool has bad data for a competitor, the analysis is bad too, no matter how well it's written up.

How to build it

If you want to replicate this:

1. Get API access to a keyword data source (Semrush, DataForSEO, or both) 2. Write a script that pulls competitor keyword sets and domain gaps for a list of domains 3. Check which competitors were actually analyzed against the list you meant to analyze 4. Dedupe, then filter hard for relevance, location and intent 5. Have a model cluster the filtered set and draft a memo 6. Review the memo before anything reaches a roadmap

The scripting takes a weekend. The filtering rules are what you'll keep refining.

Who decides

Competitive analysis is data collection plus pattern recognition, and agents are good at both as long as the data going in is clean.

The agent doesn't make strategy decisions, such as whether to move into a new niche. It surfaces patterns and flags candidates, and you decide which of those are worth pursuing.

What's next

Things I'd like to add:

1. Scheduled re-runs so changes in a competitor set show up without anyone remembering to ask 2. Page-level gap analysis: "their guide covers these subtopics and yours doesn't" 3. Search Console tie-in so gaps are ranked against what the client already gets impressions for

Once the pull and the filters are scripted, each of these gets cheaper.