Export Crawl Logs

POST /functions/v1/export-crawl-logs

Beta. This endpoint is new and its response shape may still change. Where Export Crawl Analytics returns aggregates, this endpoint returns the individual requests behind them — one row per bot hit, straight from the daily archive.

Exports the raw request log of bot traffic for one domain: every archived crawler request with its timestamp, path, user agent, verification verdict, and response metadata. Three formats:

json (default) — rows in a JSON envelope, cursor-paginated.
csv — the same rows and pagination as an RFC 4180 CSV download.
parquet — presigned download links to the daily Apache Parquet archive files themselves, ready for DuckDB, pandas, or any warehouse. No server-side decoding, no pagination — the fastest way to bulk-load history.

Always call this endpoint from a server-side environment (backend API, serverless function, build script, etc.). Never include your API key in client-side code — it will be visible to anyone inspecting your frontend.

Programmatic (API key) access requires the Pro plan and above — requests from Starter accounts return 403 plan_upgrade_required.

Recent days are provisional. A day is only frozen into the permanent archive after a settling window of a few days; until then it is served from a provisional copy and listed in provisionalDays — real rows, but tier verdicts may still upgrade. See Data freshness. Archived history is retained for ~13 months.

Windows are capped at 31 days

Raw logs are far denser than aggregates, so this endpoint accepts period values of 1d, 7d (the default), or 30d, and explicit startDate/endDate ranges spanning at most 31 days. For longer history, iterate over consecutive windows — or use format: "parquet" and download months of daily files with one request per 31-day window.

Data freshness

Every day in your window falls into exactly one of three states:

Day	Served from	How the response marks it
Today (still in progress)	Not available yet	Listed in `missingDays`
Complete but not yet settled (roughly the last 3 days)	Provisional copy, rewritten at each UTC day close	Listed in `provisionalDays`
Settled	Frozen archive — immutable	Neither list

A finished day becomes exportable shortly after the next UTC midnight. Provisional rows are the same requests you’ll see after the freeze — only the verification verdict (tier) can change, upgrading as late verification completes. If your pipeline depends on final verdicts, re-export any day that appeared in provisionalDays once it has settled.

Request

Headers

Header	Required	Description
`Authorization`	Yes	`Bearer hado_sk_your_key_here`
`Content-Type`	Yes	`application/json`

Body Parameters

Parameter	Type	Required	Description
`domainId`	string	Yes	The domain to export. Get IDs from List Domains.
`period`	string	No	Rolling window ending today: `1d`, `7d`, or `30d`. Defaults to `7d`. Cannot be combined with `startDate`/`endDate`.
`startDate`	string	No	Start of an explicit date range, `YYYY-MM-DD` (UTC, inclusive).
`endDate`	string	No	End of an explicit date range, `YYYY-MM-DD` (UTC, inclusive). Optional — defaults to today when `startDate` is supplied. The span may cover at most 31 days.
`format`	string	No	`json` (default), `csv`, or `parquet`.
`botOperator`	string \| string[]	No	Only rows whose source IP matched these bot operators (families) — e.g. `"openai"`, `"anthropic"`, `"google"`. `json`/`csv` only.
`tier`	string \| string[]	No	Only rows with these verification verdicts: `verified`, `declared`, `automation`, `unknown`. Rows with no verdict yet (`tier: null`) never match a tier filter — on provisional days that can exclude rows whose verdict simply hasn’t landed. `json`/`csv` only.
`servedFrom`	string \| string[]	No	Only rows served from: `cache`, `render`, `origin`. `json`/`csv` only.
`pathPrefix`	string	No	Only rows whose path starts with this prefix (e.g. `/blog/`). `json`/`csv` only.
`pageSize`	number	No	Rows per page, `1`–`2000`. Defaults to `500`. `json`/`csv` only.
`cursor`	string	No	Continuation token from a previous response. When set, all other parameters are taken from the cursor — it pins the window, format, filters and page size.

Filters and pageSize apply only to the json/csv formats; sending them with format: "parquet" is a 400 (parquet returns whole archive files, which are never server-filtered).

Example Request

curl -X POST https://api.hadoseo.com/functions/v1/export-crawl-logs \
  -H "Authorization: Bearer hado_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "domainId": "11111111-1111-1111-1111-111111111111",
    "period": "7d",
    "botOperator": ["openai", "anthropic"]
  }'

Response

Success (200) — `format: "json"`

{
  "generatedAt": "2026-09-09T12:00:00.000Z",
  "period": "7d",
  "startDate": "2026-09-03",
  "endDate": "2026-09-09",
  "userId": "00000000-0000-0000-0000-000000000000",
  "domain": "example.com",
  "domainId": "11111111-1111-1111-1111-111111111111",
  "role": "owner",
  "format": "json",
  "rows": [
    {
      "ts": "2026-09-03T08:14:22.271Z",
      "domain": "example.com",
      "path": "/pricing",
      "userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot",
      "uaHash": "5f2c8a01",
      "servedFrom": "cache",
      "status": 200,
      "bytes": 48211,
      "botOperator": "openai",
      "matchedRange": "20.171.0.0/16",
      "tier": "verified",
      "contentHash": "a9f5d1c0b7e28843"
    }
  ],
  "rowCount": 500,
  "missingDays": ["2026-09-09"],
  "provisionalDays": ["2026-09-07", "2026-09-08"],
  "nextCursor": "eyJ2IjoxLCJkIjoi..."
}

Row Fields

Field	Type	Description
`ts`	string	Request timestamp, ISO 8601 (UTC, millisecond precision). Rows are ordered oldest → newest.
`domain`	string	The hostname the request was served under.
`path`	string	The path the bot requested.
`userAgent`	string	The bot’s raw User-Agent string.
`uaHash`	string	Stable hash of the User-Agent, for grouping without string comparison.
`servedFrom`	string	How the response was produced: `cache`, `render`, or `origin`.
`status`	number	HTTP status returned to the bot.
`bytes`	number	Response size in bytes.
`botOperator`	string \| null	The bot operator (family) whose published IP ranges contained the request’s source IP — e.g. `openai`. `null` when the IP matched no known operator.
`matchedRange`	string \| null	The specific CIDR that matched.
`tier`	string \| null	The verification verdict: `verified`, `declared`, `automation`, or `unknown`; `null` when none was recorded. Final once the day is frozen; on `provisionalDays` it reflects verification progress as of the last day-close write and may still upgrade.
`contentHash`	string \| null	Hash of the HTML version served (joins to Export Page Snapshots). `null` for non-HTML responses.

Privacy: no IP addresses. Raw client IPs are never stored anywhere in HadoSEO. The archive keeps only facts derived from the IP before it was dropped, of which the operator/CIDR match above is what this endpoint exposes.

Envelope Fields

Field	Type	Description
`generatedAt`	string	When the export was produced.
`period`	string \| null	The rolling period applied, or `null` for an explicit range or any cursor request.
`startDate` / `endDate`	string	Resolved window (`YYYY-MM-DD`, inclusive).
`userId`	string	The account that owns the API key.
`domain` / `domainId` / `role`	string	The exported domain and your access level (`owner`, `manager`, `viewer`).
`format`	string	The delivery format of this response.
`rowCount`	number	Rows on this page. `0` with an empty `rows` array is a valid page — not a 404.
`missingDays`	string[]	Days scanned by this page with no data yet — normally just today (still in progress).
`provisionalDays`	string[]	Days scanned by this page served from the provisional (not yet frozen) copy — their `tier` values may still upgrade when the day settles.
`nextCursor`	string \| null	Pass as `cursor` to fetch the next page. `null` on the last page.

Success (200) — `format: "csv"`

The body is the CSV itself (Content-Type: text/csv), with a header row of the same field names and RFC 4180 quoting. Because there is no JSON envelope, pagination metadata moves to response headers:

Header	Description
`X-Next-Cursor`	Continuation token — absent on the last page.
`X-Missing-Days`	Comma-separated days with no data yet.
`X-Provisional-Days`	Comma-separated days served from the provisional (not yet frozen) copy.
`Content-Disposition`	Suggests a filename like `crawl-logs-example.com-2026-09-03-2026-09-09.csv`.

# Page through a CSV export, concatenating pages (strip the repeated header).
CURSOR=""
BODY='{"domainId": "11111111-1111-1111-1111-111111111111", "format": "csv", "period": "30d"}'
while :; do
  RESP_HEADERS=$(mktemp)
  curl -s -D "$RESP_HEADERS" -X POST https://api.hadoseo.com/functions/v1/export-crawl-logs \
    -H "Authorization: Bearer hado_sk_your_key_here" -H "Content-Type: application/json" \
    -d "$BODY" >> crawl-logs.csv
  CURSOR=$(grep -i '^x-next-cursor:' "$RESP_HEADERS" | tr -d '\r' | cut -d' ' -f2)
  [ -z "$CURSOR" ] && break
  BODY="{\"cursor\": \"$CURSOR\"}"
done

Success (200) — `format: "parquet"`

{
  "generatedAt": "2026-09-09T12:00:00.000Z",
  "period": "7d",
  "startDate": "2026-09-03",
  "endDate": "2026-09-09",
  "userId": "00000000-0000-0000-0000-000000000000",
  "domain": "example.com",
  "domainId": "11111111-1111-1111-1111-111111111111",
  "role": "owner",
  "format": "parquet",
  "files": [
    {
      "day": "2026-09-03",
      "part": 0,
      "url": "https://....r2.cloudflarestorage.com/...?X-Amz-Signature=...",
      "sizeBytes": 1834722,
      "expiresAt": "2026-09-09T13:00:00.000Z",
      "provisional": false
    }
  ],
  "fileCount": 6,
  "missingDays": ["2026-09-09"],
  "provisionalDays": ["2026-09-07", "2026-09-08"],
  "urlExpiresAt": "2026-09-09T13:00:00.000Z"
}

One file per day (rarely more: extremely high-traffic days split into numbered parts). Files with provisional: true come from the not-yet-frozen copy of a recent day — the frozen file that later replaces it may carry upgraded verdict_tier values. Each url is a temporary, pre-authorized link — fetch it with a plain GET, no Authorization header. The files use the parquet schema underlying the row fields above (snake_case column names: ts as epoch-millisecond INT64, ua_raw, matched_operator, verdict_tier, …) plus two archive-internal columns not present in json/csv rows: stream_id (ingest id) and ip_asn (org-level ASN of the source IP; 0 = unknown).

-- DuckDB: load a whole export in one line, then query like a table.
SELECT matched_operator, count(*), sum(bytes)
FROM read_parquet(['<url1>', '<url2>'])
GROUP BY 1 ORDER BY 2 DESC;

Treat parquet urls as secrets. Anyone with a link can download that day’s full crawl log until urlExpiresAt — about 1 hour — without authenticating. Don’t log these links or paste them anywhere public; request fresh ones instead.

Pagination

json and csv exports walk the archive oldest → newest and return pageSize rows per request. Pass the returned cursor (body nextCursor, or the X-Next-Cursor header for CSV) as cursor in the next request — by itself; the token pins everything else. Two properties worth knowing:

Resuming is cheap: the cursor records an exact file position, so page 50 costs the same as page 2.
Consistency is guaranteed: frozen archive files are write-once, and the cursor carries an integrity check — if the file behind your cursor changed you get 400 invalid_cursor rather than silently shifted rows. Restart the export. This is rare for frozen days but expected for provisional days paginated across a day close (they’re rewritten daily), so treat invalid_cursor as a normal restart signal in long-running exports.

Error Responses

Status	Body	Description
400	`{ "error": "invalid_request", "message": "..." }`	Missing `domainId`; an invalid window, `format`, filter, or `pageSize`; a window over 31 days; filters combined with `format: "parquet"`; or a page that would fetch more archive data than one request may (narrow the window, add filters, or switch to `parquet`). The `message` field explains which.
400	`{ "error": "invalid_cursor", "message": "..." }`	The `cursor` token is malformed, or the underlying archive changed since it was minted. Restart from the first page.
401	`{ "error": "invalid_api_key" }`	API key is missing, invalid, or revoked.
403	`{ "error": "plan_upgrade_required" }`	Programmatic export requires the Pro plan or above.
404	`{ "error": "domain_not_found" }`	The `domainId` doesn’t exist or isn’t accessible to your account.
429	`{ "error": "rate_limit_exceeded_monthly", "usage": 300, "limit": 300 }`	Rate limit exceeded. The code is `rate_limit_exceeded_monthly` or `rate_limit_exceeded_per_minute` — match on the `rate_limit_exceeded` prefix to catch both.

Errors return the JSON shapes above regardless of the requested format.

Examples

Which AI bots hit a section, verified only

async function verifiedAiHits(domainId) {
  const rows = [];
  let body = {
    domainId,
    period: "30d",
    tier: "verified",
    pathPrefix: "/docs/",
    pageSize: 2000,
  };
  while (body) {
    const res = await fetch("https://api.hadoseo.com/functions/v1/export-crawl-logs", {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${process.env.HADOSEO_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(body),
    });
    if (!res.ok) throw new Error(`${res.status}: ${(await res.json()).message}`);
    const data = await res.json();
    rows.push(...data.rows);
    body = data.nextCursor ? { cursor: data.nextCursor } : null;
  }
  return rows;
}

Bulk-loading history into DuckDB

curl -s -X POST https://api.hadoseo.com/functions/v1/export-crawl-logs \
  -H "Authorization: Bearer hado_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"domainId": "11111111-1111-1111-1111-111111111111", "format": "parquet", "period": "30d"}' \
  | python3 -c 'import json,sys; [print(f["url"]) for f in json.load(sys.stdin)["files"]]' \
  | xargs -I{} curl -sO {}

Each API request counts against your monthly quota. For recurring pipelines, prefer format: "parquet": one request returns links to a full month of daily files, versus potentially many paginated json requests for the same data.

Overview

Endpoints

Export Crawl Logs

Windows are capped at 31 days

Data freshness

Request

Headers

Body Parameters

Example Request

Response

Success (200) — `format: "json"`

Row Fields

Envelope Fields

Success (200) — `format: "csv"`

Success (200) — `format: "parquet"`

Error Responses

Examples

Which AI bots hit a section, verified only

Bulk-loading history into DuckDB

​Windows are capped at 31 days

​Data freshness

​Request

​Headers

​Body Parameters

​Example Request

​Response

​Success (200) — format: "json"

​Row Fields

​Envelope Fields

​Success (200) — format: "csv"

​Success (200) — format: "parquet"

​Pagination

​Error Responses

​Examples

​Which AI bots hit a section, verified only

​Bulk-loading history into DuckDB

Windows are capped at 31 days

Data freshness

Request

Headers

Body Parameters

Example Request

Response

Success (200) — `format: "json"`

Row Fields

Envelope Fields

Success (200) — `format: "csv"`

Success (200) — `format: "parquet"`

Pagination

Error Responses

Examples

Which AI bots hit a section, verified only

Bulk-loading history into DuckDB