SEO Lead
The Programmatic SEO Spy: How We Found Our Competitors' Best-Kept Secret
Key Takeaway
We crawled competitor sitemaps to detect which URL patterns are programmatic SEO, estimated traffic per pattern, and found content gaps worth thousands of monthly visits that nobody was targeting.
The Problem
The most valuable SEO insight isn't which individual pages rank — it's which systems your competitors built to rank at scale. A single page that ranks for one keyword is a data point. A /vs/ directory with 40 pages, each ranking for a " alternative" query, is a machine.
When we crawled competitor sitemaps manually, we saw the individual pages but missed the pattern. We needed an agent that could look at a sitemap and immediately identify: "This is programmatic. These URLs share a template. There are N pages here and they're collectively pulling an estimated X visits per month."
That's a different analysis from ranking keywords. It's recognizing the architecture behind competitor traffic — and then finding the gaps in that architecture that nobody has claimed.
The Solution
A sitemap crawler that classifies URL patterns algorithmically, estimates traffic per pattern, and runs a gap analysis against our own URL structure.
The Process
Step 1: Crawl the sitemaps.
The agent receives a list of competitor domains and crawls each sitemap recursively.
Step 2: Pattern detection and analysis.
Here's the prompt fed to the analysis skill:
View details
You are an SEO architecture analyst. I'm giving you a list of URLs from a competitor's sitemap.
Your job:
1. Identify URL patterns that appear to be programmatic (generated from templates + data)
2. For each pattern: name it, describe the template, count the pages
3. Estimate monthly organic traffic per pattern (use keyword volume × position × CTR estimates)
4. Identify what entity types they're targeting (tools, industries, use cases, competitors, etc.)
5. Flag patterns that appear to be working (high page count + estimated traffic)
URLs to analyze: [sitemap URLs]
Step 3: Pattern analysis output.
markdownShow code
## langchain.com — Programmatic SEO Pattern Analysis
### Pattern 1: /integrations/[tool-slug]/
- Pages: 187
- Template: Integration documentation page for each supported tool
- Example URLs: /integrations/openai/, /integrations/pinecone/, /integrations/anthropic/
- Est. avg traffic per page: ~280 visits/month
- Est. total pattern traffic: ~52,360 visits/month
- Why it works: Captures "[tool] langchain integration" searches — high commercial intent
### Pattern 2: /docs/[concept]/
- Pages: 94
- Template: Concept documentation pages
- Est. avg traffic per page: ~620 visits/month
- Est. total pattern traffic: ~58,280 visits/month
- Why it works: Dominates "how to [concept] with langchain" long-tail searches
### Pattern 3: /blog/[use-case]-with-langchain/
- Pages: 43
- Template: Tutorial articles targeting specific use case + brand
- Est. avg traffic per page: ~1,100 visits/month
- Est. total pattern traffic: ~47,300 visits/month
- Why it works: Captures researchers validating langchain for their specific use case
---
## crewai.com — Programmatic SEO Pattern Analysis
### Pattern 1: /vs/[competitor]/
- Pages: 12
- Template: Comparison landing page vs. each competitor
- Est. avg traffic per page: ~700 visits/month
- Est. total pattern traffic: ~8,400 visits/month
- Why it works: Captures high-intent "crewai vs [alternative]" searches
Gap identified: crewai has /vs/ pages for AutoGen, LangChain, and AutoGPT.
They're missing: vs/MetaGPT, vs/AgentGPT, vs/Superagent, vs/Mr.Chief
Those 4 missing /vs/ pages represent ~2,800 potential monthly visits — and we could build our
own /vs/crewai/ page targeting the same "crewai alternative" searches they haven't covered.
Step 4: Gap Matrix
markdownShow code
## Programmatic SEO Gap Analysis
| Pattern Type | langchain.com | crewai.com | autogen | pyratzlabs.com | Opportunity |
|-------------|--------------|-----------|---------|----------------|------------|
| /vs/[competitor]/ | 8 pages | 12 pages | 3 pages | 0 pages | HIGH — build /vs/ directory |
| /for-[industry]/ | 0 pages | 5 pages | 0 pages | 0 pages | HIGH — first mover opportunity |
| /integrations/[tool]/ | 187 pages | 12 pages | 0 pages | 0 pages | MEDIUM — build top 20 first |
| /alternatives-to/[tool]/ | 0 pages | 6 pages | 0 pages | 0 pages | HIGH — high commercial intent |
| /[use-case]-tutorial/ | 43 pages | 18 pages | 9 pages | 0 pages | MEDIUM — effort intensive |
Priority recommendation: Build /vs/ directory first (fastest to execute, highest conversion intent)
The Results
5
Competitors crawled
4,200+
Total URLs analyzed
18 distinct patterns
Programmatic patterns identified
/vs/ pages — ~8,400 visits/month at competitors
Highest-value gap found
34 /vs/ pages, 12 /for-industry/ pages
Pages we built from gap analysis
~6,200 visits/month
Traffic from new pages (3 months)
28 minutes
Time for full analysis
The /vs/ pages drove disproportionate results because comparison-intent traffic converts. Someone searching "crewai vs mrchief" is actively evaluating. They're closer to a decision than someone searching "what is a multi-agent system."
Your competitors have already done the expensive A/B test on what programmatic SEO patterns work. Their sitemaps are the report — you just need to know how to read them.
Related case studies
Studio Founder
Reverse-Engineering Competitor SEO Without Paying for Semrush
We built a complete competitive SEO intelligence pipeline — domain authority, keyword gaps, backlink profiles, traffic-driving URL patterns — using free Apify actors and three Mr.Chief skills. Zero paid subscriptions.
SEO Lead
Building 200+ SEO Pages from 3 Templates
We used an AI agent to design 3 programmatic SEO templates that generated 214 unique, high-quality pages targeting real search demand — for $38.50 total.
SEO Analyst
Our Competitor's SEO Strategy in 30 Minutes — Without Semrush
We reverse-engineered our competitors' full SEO strategies — domain authority, keywords, backlinks, traffic estimates — without paying for Semrush or Ahrefs, using Apify actors that scrape their public pages.
Want results like these?
Start free with your own AI team. No credit card required.