DataForSEO has the cleanest SERP data I've found. There's no UI to work around, only API calls and JSON responses.

This tutorial assumes you can make HTTP requests. If you can't, skip to the "Why this matters" section.

Why DataForSEO instead of SEMrush or Ahrefs

SEMrush is better for UI-based competitor analysis. Ahrefs is better for backlink research. DataForSEO is better for programmatic keyword research at scale.

If you're building an agent or a custom dashboard, you need clean API access. SEMrush and Ahrefs have APIs, but DataForSEO's is more straightforward and cheaper.

Setup

1. Sign up at dataforseo.com 2. Go to your account settings and grab your API credentials (username and password) 3. Your endpoint is always https://api.dataforseo.com/v3/

There's no OAuth flow to set up.

Your first query: Get SERP data for a keyword

curl -s -u "username:password" \
  -H "Content-Type: application/json" \
  -X POST \
  https://api.dataforseo.com/v3/serp/google/organic/live/advanced \
  -d '[{
    "keyword": "best travel credit card",
    "location_code": 2840,
    "language_code": "en",
    "depth": 100
  }]' | jq '.'

What this does:

The response includes:

Note what's not in there: traffic estimates and authority scores. Those come from other DataForSEO endpoints (the Labs and Backlinks APIs). Billing is per request and scales with depth and mode, so check their pricing page for current rates. For a single keyword the cost is small.

Your second query: Get keyword metrics

curl -s -u "username:password" \
  -H "Content-Type: application/json" \
  -X POST \
  https://api.dataforseo.com/v3/keywords_data/google_ads/search_volume/live \
  -d '[{
    "keywords": ["best travel credit card", "travel rewards card", "credit card for travel"],
    "location_code": 2840,
    "language_code": "en"
  }]' | jq '.'

Response includes:

This is Google Ads data, the same source behind Keyword Planner. It's still a rounded monthly average rather than an exact count, so treat it as a strong signal.

Find keyword gaps (the useful part)

This bash script finds keywords your competitors rank for that you don't:

#!/bin/bash

# Get the domains ranking organically for a seed keyword
curl -s -u "username:password" \
  -H "Content-Type: application/json" \
  -X POST \
  https://api.dataforseo.com/v3/serp/google/organic/live/advanced \
  -d '[{ "keyword": "travel credit card", "location_code": 2840, "language_code": "en" }]' \
  | jq -r '.tasks[0].result[0].items[] | select(.type == "organic") | .domain' \
  | sort -u > competitor_domains.txt

# For each domain, pull the keywords it ranks for
while IFS= read -r domain; do
  curl -s -u "username:password" \
    -H "Content-Type: application/json" \
    -X POST \
    https://api.dataforseo.com/v3/dataforseo_labs/google/ranked_keywords/live \
    -d "[{\"target\": \"$domain\", \"location_code\": 2840, \"language_code\": \"en\", \"limit\": 100}]" \
    | jq -c '.tasks[0].result[0].items[] | {keyword: .keyword_data.keyword, search_volume: .keyword_data.keyword_info.search_volume, competition: .keyword_data.keyword_info.competition}' \
    >> competitor_keywords.jsonl
done < competitor_domains.txt

# Keep keywords with real volume and lower competition
jq -c 'select(.search_volume != null and .search_volume > 100 and .competition != null and .competition < 0.5)' competitor_keywords.jsonl \
  | head -20

The output is low-competition keywords that competitors already rank for, which gives you a working view of their strategy.

Batch queries

#!/bin/bash

KEYWORDS=("travel credit card" "best credit card" "rewards card" "no annual fee card" "cashback credit card")

for keyword in "${KEYWORDS[@]}"; do
  curl -s -u "username:password" \
    -H "Content-Type: application/json" \
    -X POST \
    https://api.dataforseo.com/v3/serp/google/organic/live/advanced \
    -d "[{ \"keyword\": \"$keyword\", \"location_code\": 2840, \"language_code\": \"en\" }]" \
    | jq -c '.tasks[0].result[0].items[]' >> all_keywords_data.json

  sleep 1  # Be respectful with API calls
done

# Aggregate and analyze
cat all_keywords_data.json | jq -s 'group_by(.type) | map({type: .[0].type, count: length})'

This gives you SERP data for multiple keywords in one go, and the aggregate shows which SERP features keep showing up across your niche. That's hundreds of data points for very little money.

Practical example: Find content opportunities

#!/bin/bash

# Query keyword data for your target niche
curl -s -u "username:password" \
  -H "Content-Type: application/json" \
  -X POST \
  https://api.dataforseo.com/v3/keywords_data/google_ads/search_volume/live \
  -d '[{ "keywords": ["seo for beginners", "seo tutorial", "how to do seo", "seo tips"], "location_code": 2840, "language_code": "en" }]' \
  | jq '.tasks[0].result[] | select(.search_volume > 50 and .search_volume < 500) | {keyword, search_volume}'

This finds keywords with real search volume, but not so much that they're impossible to rank for. That range is the sweet spot for content that can actually bring in traffic.

Real numbers from my workflow

At the agency, our keyword research pipeline runs on DataForSEO. Two real runs for Regulator Marine:

The output is a keyword universe, a cluster summary and a content roadmap, which feeds the content calendar.

Building a full keyword research system

What I've built on top of DataForSEO, and what it taught me:

A keyword research pipeline:

# Seed terms -> related keywords + keyword ideas
# -> search volume and difficulty
# -> SERP validation on a capped sample
# -> clusters -> content roadmap (CSV)

Two lessons from running it. First, respect batch limits: one endpoint caps a request at 1,000 keywords, and going over didn't throw an error; it silently returned empty volume and difficulty. Size each cycle so it stays under the cap. Second, don't trust your own call counter for spend, because cached results get counted as calls. Read your account balance before and after a run (the appendix/user_data endpoint is free) and use that as the real cost.

A rank tracker: It combines Google Search Console, Semrush and DataForSEO, with cost caps built in. Verification scans cost between $0.37 and $1.67 each. I run it on demand when there's a question to answer, rather than on a schedule.

Why this matters

You don't need a $300/month dashboard to understand your keyword space. You need: 1. Clean API access to SERP data 2. The ability to make HTTP requests (bash, Python, whatever) 3. 15 minutes to write a script

DataForSEO is pay-as-you-go, so you pay for the calls you make instead of a seat. The runs above cost a couple of dollars each.

A lot of what's sold as a "keyword research tool" is UI on top of data you can query directly for much less.

Subscription platforms sell convenience and a dashboard, which is worth paying for if you need it. The insight still comes from the questions you ask of the data.

Going straight to the API takes a bit of technical skill, but the savings are real, and your analysis isn't limited by someone else's dashboard design.