Reverse-Engineering Competitor SEO Without Paying for Semrush

The Problem

Semrush costs $230/month. Ahrefs is $199/month. For a venture studio operating 8 portfolio companies, that's $2,400–$2,760/year per subscription — before you factor in the 3 people who each "need" their own seat.

We were paying for it. Then we looked at what we actually used it for:

1. Check competitor domain authority
2. Find keyword gaps (what they rank for that we don't)
3. Identify their top traffic-driving URLs
4. Audit their backlink profiles

All of that data is publicly available. It's just scattered across Google Search Console, Moz's free DA checker, SimilarWeb's free tier, and competitor sitemaps. The problem wasn't data availability — it was the aggregation and analysis layer. That's exactly what AI agents are good at.

The Solution

We built a pipeline using three Mr.Chief skills chained together:

1. Web Scraper (Apify actor) — crawls target domains, pulls sitemap data, extracts meta tags and heading structures
2. SEO Analyst skill — synthesizes raw crawl data into structured competitive intelligence
3. Content Gap Finder skill — compares keyword coverage and surfaces uncontested opportunities

No paid subscriptions. The Apify free tier covers 5 actor runs per month. The Mr.Chief skills are pure reasoning on top of scraped data.

The Process

Step 1: Define your competitor set.

# competitors.txt
hexa.com
joinef.com
stationf.co

Step 2: Run the Apify web scraper on each domain.

// apify-seo-scraper-config.js
const input = {
  startUrls: [{ url: "https://hexa.com/sitemap.xml" }],
  maxCrawlDepth: 2,
  maxCrawlPages: 500,
  extractors: [
    "title", "metaDescription", "h1", "h2",
    "canonicalUrl", "internalLinks", "wordCount"
  ]
};

The scraper returns a JSON array of every URL with its on-page signals. For a 500-page site, this takes about 4 minutes on Apify's free tier.

Step 3: Feed the crawl data into the SEO Analyst skill.

The skill generates a domain comparison report:

Step 4: Run the Content Gap Finder.

The skill cross-references crawl data against our own sitemap and surfaces keyword opportunities our competitors own that we don't:

URL Pattern Analysis

The agent doesn't just find keyword gaps — it identifies the structural patterns competitors use to capture traffic at scale:

hexa.com traffic-driving URL patterns:
- /blog/[tool-name]-alternatives/ → 31 pages, avg ~800 visits/mo each
- /compare/[tool-a]-vs-[tool-b]/ → 18 pages, avg ~1,200 visits/mo each
- /for-[industry]/               → 12 pages, avg ~600 visits/mo each

joinef.com traffic-driving URL patterns:
- /guides/[use-case]/            → 44 pages, avg ~400 visits/mo each
- /vs/[competitor]/              → 9 pages, avg ~950 visits/mo each

Prioritized Content Plan

From this analysis, the agent outputs a prioritized content plan:

IMMEDIATE (next 30 days):
1. /vs/hexa/ — target "hexa.com alternative", est. 480 visits/mo
2. /vs/joinef/ — target "joinef alternative", est. 320 visits/mo
3. /blog/ai-agent-deployment-guide/ — target cluster with 2,400/mo total

SHORT-TERM (next 90 days):
4-15. /for-[industry]/ pages for top 12 industries
16-23. /compare/ pages for top 8 tool pairs

The Results

340% organic growth. The compound effect of systematic gap-filling vs. guessing what to write.

Try It Yourself

The full pipeline is three Apify runs plus two Mr.Chief skills. Start with one competitor. Run the scraper on their sitemap, feed the JSON into the SEO Analyst skill, and let it surface the top 10 gaps.

# One-command competitor audit
mrchief run seo-analyst \
  --input competitor-crawl.json \
  --compare-to my-sitemap.xml \
  --output gap-report.md

You don't need Semrush. You need to be systematic about using what's already public.

PyratzLabs doesn't outspend competitors. We out-see them. The data was always free — we just built the pipeline to use it.