SEO Analyst
Our Competitor's SEO Strategy in 30 Minutes — Without Semrush
Key Takeaway
We reverse-engineered our competitors' full SEO strategies — domain authority, keywords, backlinks, traffic estimates — without paying for Semrush or Ahrefs, using Apify actors that scrape their public pages.
The Problem
Competitor SEO analysis is one of those tasks that sounds expensive because the tools that do it are expensive. Semrush: $230/month. Ahrefs: $199/month. Moz Pro: $179/month. The tools position themselves as indispensable.
They're not. They're aggregators of public signals:
- Domain authority is calculable from publicly accessible link data
- Keyword rankings are visible from Google Search Console for your own site — and inferrable for competitors from their page content and title tags
- Traffic estimates are derived from keyword ranking × search volume × estimated CTR curves
- Backlink profiles are available from Common Crawl and Moz's free DA checker
Everything a $230/month tool shows you is either public or mathematically estimated. The edge is in automation and aggregation — which is exactly what Apify + Mr.Chief provides for a fraction of the cost.
The Solution
A two-step pipeline:
- Apify scrapes competitor domains — sitemaps, on-page signals, backlink proxies
- Mr.Chief SEO Analyst synthesizes raw data into competitive intelligence reports
Total infrastructure cost: ~$0.80 per competitor in Apify usage + ~$0.20 in LLM tokens.
The Process
Step 1: Run the competitor crawler.
javascriptShow code
// competitor-seo-crawler.js
const input = {
startUrls: [
{ url: "https://langchain.com/sitemap.xml" },
{ url: "https://crewai.com/sitemap.xml" },
{ url: "https://autogen.microsoft.com/sitemap.xml" }
],
maxCrawlPages: 500,
extractors: [
"pageTitle", "metaDescription", "h1", "h2s",
"wordCount", "internalLinks", "externalLinks",
"canonicalUrl", "publishDate", "modifiedDate",
"structuredData", "openGraphTags"
],
respectRobotsTxt: true,
maxConcurrency: 5
};
Step 2: Domain Comparison
The SEO Analyst skill produces a structured competitive overview:
markdownShow code
## Domain Comparison Report
| Domain | Domain Authority | Est. Monthly Organic | Ranking Keywords | Backlinks |
|--------|-----------------|---------------------|------------------|-----------|
| langchain.com | 71 | ~82,000 | 5,400+ | 31,200 |
| crewai.com | 48 | ~22,000 | 1,600+ | 7,800 |
| autogen.microsoft.com | 83 | ~41,000 | 2,900+ | 89,000 |
| pyratzlabs.com | 34 | ~2,100 | 380 | 1,200 |
Note: DA estimates from Moz free tier; traffic estimates from keyword volume × CTR model
Step 3: Keyword Gap Analysis
markdownShow code
## Keyword Gap Analysis
| Keyword | Top Competitor | Est. Volume | Their Position | Our Position | Gap Type |
|---------|---------------|-------------|----------------|--------------|---------|
| "build AI agent" | langchain.com | 4,400/mo | #2 | Not ranking | Content gap |
| "agent framework python" | crewai.com | 2,900/mo | #3 | Not ranking | Content gap |
| "AutoGPT alternative" | crewai.com | 1,800/mo | #1 | Not ranking | Quick win |
| "deploy AI agents" | langchain.com | 1,600/mo | #4 | #31 | Page 1 opportunity |
| "multi-agent system tutorial" | crewai.com | 1,200/mo | #2 | Not ranking | Content gap |
| "agent orchestration" | autogen | 880/mo | #1 | Not ranking | Content gap |
Step 4: Competitor Top Pages
markdownShow code
## Competitor Top Traffic-Driving Pages (Estimated)
### langchain.com
| URL | Est. Monthly Traffic | Primary Keyword |
|-----|---------------------|-----------------|
| /docs/get_started/quickstart | ~8,200 | "langchain tutorial" |
| /blog/tool-calling | ~4,600 | "agent tool calling" |
| /docs/concepts/agents | ~3,900 | "what is a langchain agent" |
### crewai.com
| URL | Est. Monthly Traffic | Primary Keyword |
|-----|---------------------|-----------------|
| /blog/getting-started | ~3,400 | "crewai tutorial" |
| /vs/autogen | ~2,100 | "crewai vs autogen" |
| /for-enterprises | ~1,800 | "enterprise AI agents" |
The Results
| Metric | Semrush | Our Pipeline |
|---|---|---|
| Monthly cost | $230 | ~$2.40 per full audit |
| Setup time | 5 minutes | 45 minutes (one-time) |
| Competitors covered | Unlimited (paid plan) | Unlimited |
| Data freshness | Updated regularly | On-demand |
| Customization | Limited to their UI | Full control |
| Output format | Web UI + CSV | Markdown, JSON, Slack |
| First audit date to action | Same day | Same day |
We run a full competitive audit monthly for $2.40 total. For four competitors, that's $28.80/year versus $2,760/year for a Semrush team plan.
The data quality difference: minimal on the metrics that matter (keyword rankings and traffic estimates are estimates either way). The workflow flexibility advantage: significant — our pipeline outputs directly into our content planning templates.
Semrush is a great product. We just don't need to pay for it — the underlying data is public, and the analysis layer is where AI agents earn their keep.
Related case studies
Studio Founder
Reverse-Engineering Competitor SEO Without Paying for Semrush
We built a complete competitive SEO intelligence pipeline — domain authority, keyword gaps, backlink profiles, traffic-driving URL patterns — using free Apify actors and three Mr.Chief skills. Zero paid subscriptions.
SEO Lead
The Programmatic SEO Spy: How We Found Our Competitors' Best-Kept Secret
We crawled competitor sitemaps to detect which URL patterns are programmatic SEO, estimated traffic per pattern, and found content gaps worth thousands of monthly visits.
SEO Lead
Building 200+ SEO Pages from 3 Templates
We used an AI agent to design 3 programmatic SEO templates that generated 214 unique, high-quality pages targeting real search demand — for $38.50 total.
Want results like these?
Start free with your own AI team. No credit card required.