Export Crawl Analytics

POST /functions/v1/export-crawl-analytics

Beta. This endpoint is new and its response shape may still change. It is the crawl-intelligence sibling of Export Analytics: same authentication, windowing, and domain scoping, but it reports attribution tiers (was the bot’s identity IP-verified?) and each bot’s intent.

Returns a single JSON document of crawl rollups for every domain on your account — both domains you own and domains shared with you. For each domain it breaks bot traffic down by bot and by attribution tier (verified / declared / unknown), and annotates every bot with its intent (why it crawls — e.g. training, search-classic, answer-fetch).

Always call this endpoint from a server-side environment (backend API, serverless function, build script, etc.). Never include your API key in client-side code — it will be visible to anyone inspecting your frontend.

Programmatic (API key) access requires the Pro plan and above — requests from Starter accounts return 403 plan_upgrade_required.

Request

Headers

Header	Required	Description
`Authorization`	Yes	`Bearer hado_sk_your_key_here`
`Content-Type`	Yes	`application/json`

Body Parameters

Choose the reporting window in one of two ways — a rolling period, or an explicit startDate/endDate range. The two are mutually exclusive.

Parameter	Type	Required	Description
`period`	string	No	Rolling window ending today: `1d`, `7d`, `30d`, `90d`, or `365d`. Defaults to `30d`. Cannot be combined with `startDate`/`endDate`.
`startDate`	string	No	Start of an explicit date range, `YYYY-MM-DD` (UTC, inclusive).
`endDate`	string	No	End of an explicit date range, `YYYY-MM-DD` (UTC, inclusive). Optional — defaults to today when `startDate` is supplied. Cannot be sent without `startDate`.
`domainId`	string	No	Restrict the export to a single domain by its ID. Omit to export every domain your account can access (the default). Get IDs from List Domains.

Range rules — a request that breaks any of these returns 400 invalid_request:

period and startDate/endDate cannot be combined.
endDate cannot be sent on its own — it requires a startDate.
Dates must be real calendar dates in YYYY-MM-DD form (e.g. 2026-02-30 is rejected).
startDate must not be after endDate, and neither may be in the future.
The range may span at most 365 days (inclusive), the same cap as the 365d period.

Example Request

curl -X POST https://api.hadoseo.com/functions/v1/export-crawl-analytics \
  -H "Authorization: Bearer hado_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "period": "30d"
  }'

Single domain, explicit range:

curl -X POST https://api.hadoseo.com/functions/v1/export-crawl-analytics \
  -H "Authorization: Bearer hado_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "domainId": "11111111-1111-1111-1111-111111111111",
    "startDate": "2026-07-01",
    "endDate": "2026-07-31"
  }'

Response

Success (200)

{
  "generatedAt": "2026-07-31T12:00:00.000Z",
  "period": "30d",
  "startDate": "2026-07-02",
  "endDate": "2026-07-31",
  "userId": "00000000-0000-0000-0000-000000000000",
  "legacyCutover": "2026-07-09",
  "domainCount": 1,
  "domains": [
    {
      "domain": "example.com",
      "domainId": "11111111-1111-1111-1111-111111111111",
      "role": "owner",
      "summary": {
        "totalCrawls": 1840,
        "namedCrawls": 1620,
        "verifiedCrawls": 1400
      },
      "botTotals": {
        "googlebot": { "intent": "search-classic", "verified": 900, "declared": 20, "unknown": 0, "total": 920 },
        "bingbot":   { "intent": "search-classic", "verified": 210, "declared": 4,  "unknown": 0, "total": 214 },
        "gptbot":    { "intent": "training",        "verified": 80,  "declared": 8,  "unknown": 0, "total": 88 },
        "unknown":   { "intent": "other",           "verified": 0,   "declared": 0,  "unknown": 220, "total": 220 }
      },
      "botSeries": [
        {
          "date": "2026-07-30",
          "googlebot": { "verified": 40, "declared": 1, "unknown": 0, "total": 41 },
          "gptbot":    { "verified": 3,  "declared": 0, "unknown": 0, "total": 3 }
        }
      ]
    }
  ]
}

Fields

Field	Type	Description
`generatedAt`	string	ISO 8601 timestamp of when the export was produced.
`period`	string \| null	The rolling period applied, or `null` when an explicit `startDate`/`endDate` range was requested.
`startDate`	string	Resolved start of the window (`YYYY-MM-DD`, inclusive).
`endDate`	string	Resolved end of the window (`YYYY-MM-DD`, inclusive).
`userId`	string	The account that owns the API key.
`legacyCutover`	string	The date on/after which data comes from verified crawl tracking. See Legacy data.
`domainCount`	number	Number of domains included.
`domains[].domain`	string	The domain name.
`domains[].domainId`	string	The domain’s unique ID.
`domains[].role`	string	Access level for this domain: `owner`, `manager`, or `viewer`.
`domains[].summary`	object	Period totals: `totalCrawls` (all crawls), `namedCrawls` (identified bots), `verifiedCrawls` (IP-verified). Always `verifiedCrawls ≤ namedCrawls ≤ totalCrawls`.
`domains[].botTotals`	object	Keyed by bot. Each value has `intent` plus the tier split (`verified`, `declared`, `unknown`) and their `total`.
`domains[].botSeries`	array	Per-day breakdown; each entry has a `date` plus, under each bot key, that day’s tier split (`verified`, `declared`, `unknown`, `total`).

Attribution tiers

Every crawl is placed in exactly one tier, based on whether the bot’s claimed identity could be confirmed:

Tier	Meaning
`verified`	The User-Agent matched a known bot and the source IP forward-confirmed via reverse DNS or the operator’s published IP ranges.
`declared`	The User-Agent claims a known bot, but the IP didn’t confirm (or the bot has no verification method).
`unknown`	The User-Agent matched no known bot. Reported under the `unknown` bot key.

verified is point-in-time: it means confirmed against the operator’s published ranges as of when those ranges were last fetched (refreshed weekly). A named bot appearing only as declared is normal — many bots (social crawlers, most AI answer bots) publish no verifiable IP ranges.

Bot intent

intent describes why a bot crawls, from our bot taxonomy:

Intent	Examples
`training`	GPTBot, ClaudeBot
`search-index`	PerplexityBot, OAI-SearchBot
`answer-fetch`	ChatGPT-User, Claude-User
`search-classic`	Googlebot, Bingbot
`social`	Twitterbot, LinkedInBot
`seo-tool`	AhrefsBot, SemrushBot
`ads`	AdsBot-Google
`other`	Verification/feed fetchers, and the `unknown` bucket

Legacy data before the cutover

Verified crawl tracking begins at legacyCutover (2026-07-09). Requested days before it are backfilled from legacy User-Agent-only analytics, so the tier semantics differ:

Only Googlebot and Bingbot were IP-verified before the cutover, so only they carry a verified tier on legacy days. Every other bot is declared (it was UA-only).
AI bots (GPTBot, ClaudeBot, PerplexityBot) were UA-only before the cutover, so they appear as declared there and can appear as verified after — expect a step-change at the cutover date, not a real traffic shift.
A few legacy columns are family aggregates (e.g. all OpenAI crawlers were counted together) and are reported under a single representative bot for those days.

For a clean, tier-accurate comparison, request windows that start on or after 2026-07-09.

Error Responses

Status	Body	Description
400	`{ "error": "invalid_request", "message": "..." }`	Invalid `period`; a malformed, out-of-order, or future date; a range over 365 days; `period` combined with a date range; or `endDate` without `startDate`. The `message` field explains which.
401	`{ "error": "invalid_api_key" }`	API key is missing, invalid, or revoked.
403	`{ "error": "plan_upgrade_required" }`	Programmatic export requires the Pro plan or above.
404	`{ "error": "domain_not_found" }`	The `domainId` doesn’t exist or isn’t accessible to your account.
429	`{ "error": "rate_limit_exceeded", "usage": 300, "limit": 300 }`	Monthly or per-minute rate limit exceeded.

Examples

JavaScript / Node.js

const response = await fetch(
  "https://api.hadoseo.com/functions/v1/export-crawl-analytics",
  {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.HADOSEO_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ period: "30d" }),
  },
);

const report = await response.json();
for (const domain of report.domains) {
  const gpt = domain.botTotals.gptbot;
  if (gpt) {
    console.log(`${domain.domain}: ${gpt.verified} verified / ${gpt.total} total GPTBot crawls`);
  }
}

Overview

Endpoints

Export Crawl Analytics

Request

Headers

Body Parameters

Example Request

Response

Success (200)

Fields

Attribution tiers

Bot intent

Legacy data before the cutover

Error Responses

Examples

JavaScript / Node.js

​Request

​Headers

​Body Parameters

​Example Request

​Response

​Success (200)

​Fields

​Attribution tiers

​Bot intent

​Legacy data before the cutover

​Error Responses

​Examples

​JavaScript / Node.js

Request

Headers

Body Parameters

Example Request

Response

Success (200)

Fields

Attribution tiers

Bot intent

Legacy data before the cutover

Error Responses

Examples

JavaScript / Node.js