> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hadoseo.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Export Crawl Logs
> Export the raw per-request bot crawl log for a domain as JSON, CSV, or the original parquet files
```
POST /functions/v1/export-crawl-logs
```
**Beta.** This endpoint is new and its response shape may still change. Where
[Export Crawl Analytics](/api-reference/endpoint/export-crawl-analytics)
returns *aggregates*, this endpoint returns the **individual requests** behind
them — one row per bot hit, straight from the daily archive.
Exports the raw request log of bot traffic for **one domain**: every archived crawler request with its timestamp, path, user agent, verification verdict, and response metadata. Three formats:
* **`json`** (default) — rows in a JSON envelope, cursor-paginated.
* **`csv`** — the same rows and pagination as an RFC 4180 CSV download.
* **`parquet`** — presigned download links to the daily [Apache Parquet](https://parquet.apache.org/) archive files themselves, ready for DuckDB, pandas, or any warehouse. No server-side decoding, no pagination — the fastest way to bulk-load history.
Always call this endpoint from a **server-side environment** (backend API, serverless function, build script, etc.). Never include your API key in client-side code — it will be visible to anyone inspecting your frontend.
Programmatic (API key) access requires the **Pro plan and above** — requests
from Starter accounts return `403 plan_upgrade_required`.
**Recent days are provisional.** A day is only *frozen* into the permanent
archive after a settling window of a few days; until then it is served from a
provisional copy and listed in `provisionalDays` — real rows, but `tier`
verdicts may still upgrade. See [Data freshness](#data-freshness). Archived
history is retained for \~13 months.
## Windows are capped at 31 days
Raw logs are far denser than aggregates, so this endpoint accepts `period` values of `1d`, `7d` (the default), or `30d`, and explicit `startDate`/`endDate` ranges spanning at most **31 days**. For longer history, iterate over consecutive windows — or use `format: "parquet"` and download months of daily files with one request per 31-day window.
## Data freshness
Every day in your window falls into exactly one of three states:
| Day | Served from | How the response marks it |
| ------------------------------------------------------ | ------------------------------------------------- | --------------------------- |
| Today (still in progress) | Not available yet | Listed in `missingDays` |
| Complete but not yet settled (roughly the last 3 days) | Provisional copy, rewritten at each UTC day close | Listed in `provisionalDays` |
| Settled | Frozen archive — immutable | Neither list |
A finished day becomes exportable shortly after the next UTC midnight. Provisional rows are the same requests you'll see after the freeze — only the verification verdict (`tier`) can change, upgrading as late verification completes. If your pipeline depends on final verdicts, re-export any day that appeared in `provisionalDays` once it has settled.
## Request
### Headers
| Header | Required | Description |
| --------------- | -------- | ------------------------------ |
| `Authorization` | Yes | `Bearer hado_sk_your_key_here` |
| `Content-Type` | Yes | `application/json` |
### Body Parameters
| Parameter | Type | Required | Description |
| ------------- | ------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `domainId` | string | **Yes** | The domain to export. Get IDs from [List Domains](/api-reference/endpoint/list-domains). |
| `period` | string | No | Rolling window ending today: `1d`, `7d`, or `30d`. Defaults to `7d`. **Cannot** be combined with `startDate`/`endDate`. |
| `startDate` | string | No | Start of an explicit date range, `YYYY-MM-DD` (UTC, inclusive). |
| `endDate` | string | No | End of an explicit date range, `YYYY-MM-DD` (UTC, inclusive). Optional — defaults to **today** when `startDate` is supplied. The span may cover at most **31 days**. |
| `format` | string | No | `json` (default), `csv`, or `parquet`. |
| `botOperator` | string \| string\[] | No | Only rows whose source IP matched these bot **operators** (families) — e.g. `"openai"`, `"anthropic"`, `"google"`. `json`/`csv` only. |
| `tier` | string \| string\[] | No | Only rows with these verification verdicts: `verified`, `declared`, `automation`, `unknown`. Rows with no verdict yet (`tier: null`) never match a tier filter — on provisional days that can exclude rows whose verdict simply hasn't landed. `json`/`csv` only. |
| `servedFrom` | string \| string\[] | No | Only rows served from: `cache`, `render`, `origin`. `json`/`csv` only. |
| `pathPrefix` | string | No | Only rows whose path starts with this prefix (e.g. `/blog/`). `json`/`csv` only. |
| `pageSize` | number | No | Rows per page, `1`–`2000`. Defaults to `500`. `json`/`csv` only. |
| `cursor` | string | No | Continuation token from a previous response. When set, **all other parameters are taken from the cursor** — it pins the window, format, filters and page size. |
Filters and `pageSize` apply only to the `json`/`csv` formats; sending them with `format: "parquet"` is a `400` (parquet returns whole archive files, which are never server-filtered).
### Example Request
```bash theme={null}
curl -X POST https://api.hadoseo.com/functions/v1/export-crawl-logs \
-H "Authorization: Bearer hado_sk_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"domainId": "11111111-1111-1111-1111-111111111111",
"period": "7d",
"botOperator": ["openai", "anthropic"]
}'
```
## Response
### Success (200) — `format: "json"`
```json theme={null}
{
"generatedAt": "2026-09-09T12:00:00.000Z",
"period": "7d",
"startDate": "2026-09-03",
"endDate": "2026-09-09",
"userId": "00000000-0000-0000-0000-000000000000",
"domain": "example.com",
"domainId": "11111111-1111-1111-1111-111111111111",
"role": "owner",
"format": "json",
"rows": [
{
"ts": "2026-09-03T08:14:22.271Z",
"domain": "example.com",
"path": "/pricing",
"userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot",
"uaHash": "5f2c8a01",
"servedFrom": "cache",
"status": 200,
"bytes": 48211,
"botOperator": "openai",
"matchedRange": "20.171.0.0/16",
"tier": "verified",
"contentHash": "a9f5d1c0b7e28843"
}
],
"rowCount": 500,
"missingDays": ["2026-09-09"],
"provisionalDays": ["2026-09-07", "2026-09-08"],
"nextCursor": "eyJ2IjoxLCJkIjoi..."
}
```
### Row Fields
| Field | Type | Description |
| -------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ts` | string | Request timestamp, ISO 8601 (UTC, millisecond precision). Rows are ordered oldest → newest. |
| `domain` | string | The hostname the request was served under. |
| `path` | string | The path the bot requested. |
| `userAgent` | string | The bot's raw User-Agent string. |
| `uaHash` | string | Stable hash of the User-Agent, for grouping without string comparison. |
| `servedFrom` | string | How the response was produced: `cache`, `render`, or `origin`. |
| `status` | number | HTTP status returned to the bot. |
| `bytes` | number | Response size in bytes. |
| `botOperator` | string \| null | The bot **operator** (family) whose published IP ranges contained the request's source IP — e.g. `openai`. `null` when the IP matched no known operator. |
| `matchedRange` | string \| null | The specific CIDR that matched. |
| `tier` | string \| null | The verification verdict: `verified`, `declared`, `automation`, or `unknown`; `null` when none was recorded. Final once the day is frozen; on `provisionalDays` it reflects verification progress as of the last day-close write and may still upgrade. |
| `contentHash` | string \| null | Hash of the HTML version served (joins to [Export Page Snapshots](/api-reference/endpoint/export-page-snapshots)). `null` for non-HTML responses. |
**Privacy: no IP addresses.** Raw client IPs are never stored anywhere in
HadoSEO. The archive keeps only facts *derived* from the IP before it was
dropped, of which the operator/CIDR match above is what this endpoint
exposes.
### Envelope Fields
| Field | Type | Description |
| ------------------------------ | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `generatedAt` | string | When the export was produced. |
| `period` | string \| null | The rolling period applied, or `null` for an explicit range **or any cursor request**. |
| `startDate` / `endDate` | string | Resolved window (`YYYY-MM-DD`, inclusive). |
| `userId` | string | The account that owns the API key. |
| `domain` / `domainId` / `role` | string | The exported domain and your access level (`owner`, `manager`, `viewer`). |
| `format` | string | The delivery format of this response. |
| `rowCount` | number | Rows on this page. `0` with an empty `rows` array is a valid page — not a 404. |
| `missingDays` | string\[] | Days **scanned by this page** with no data yet — normally just today (still in progress). |
| `provisionalDays` | string\[] | Days **scanned by this page** served from the provisional (not yet frozen) copy — their `tier` values may still upgrade when the day settles. |
| `nextCursor` | string \| null | Pass as `cursor` to fetch the next page. `null` on the last page. |
### Success (200) — `format: "csv"`
The body is the CSV itself (`Content-Type: text/csv`), with a header row of the same field names and RFC 4180 quoting. Because there is no JSON envelope, pagination metadata moves to **response headers**:
| Header | Description |
| --------------------- | ---------------------------------------------------------------------------- |
| `X-Next-Cursor` | Continuation token — absent on the last page. |
| `X-Missing-Days` | Comma-separated days with no data yet. |
| `X-Provisional-Days` | Comma-separated days served from the provisional (not yet frozen) copy. |
| `Content-Disposition` | Suggests a filename like `crawl-logs-example.com-2026-09-03-2026-09-09.csv`. |
```bash theme={null}
# Page through a CSV export, concatenating pages (strip the repeated header).
CURSOR=""
BODY='{"domainId": "11111111-1111-1111-1111-111111111111", "format": "csv", "period": "30d"}'
while :; do
RESP_HEADERS=$(mktemp)
curl -s -D "$RESP_HEADERS" -X POST https://api.hadoseo.com/functions/v1/export-crawl-logs \
-H "Authorization: Bearer hado_sk_your_key_here" -H "Content-Type: application/json" \
-d "$BODY" >> crawl-logs.csv
CURSOR=$(grep -i '^x-next-cursor:' "$RESP_HEADERS" | tr -d '\r' | cut -d' ' -f2)
[ -z "$CURSOR" ] && break
BODY="{\"cursor\": \"$CURSOR\"}"
done
```
### Success (200) — `format: "parquet"`
```json theme={null}
{
"generatedAt": "2026-09-09T12:00:00.000Z",
"period": "7d",
"startDate": "2026-09-03",
"endDate": "2026-09-09",
"userId": "00000000-0000-0000-0000-000000000000",
"domain": "example.com",
"domainId": "11111111-1111-1111-1111-111111111111",
"role": "owner",
"format": "parquet",
"files": [
{
"day": "2026-09-03",
"part": 0,
"url": "https://....r2.cloudflarestorage.com/...?X-Amz-Signature=...",
"sizeBytes": 1834722,
"expiresAt": "2026-09-09T13:00:00.000Z",
"provisional": false
}
],
"fileCount": 6,
"missingDays": ["2026-09-09"],
"provisionalDays": ["2026-09-07", "2026-09-08"],
"urlExpiresAt": "2026-09-09T13:00:00.000Z"
}
```
One file per day (rarely more: extremely high-traffic days split into numbered `part`s). Files with `provisional: true` come from the not-yet-frozen copy of a recent day — the frozen file that later replaces it may carry upgraded `verdict_tier` values. Each `url` is a temporary, pre-authorized link — fetch it with a plain `GET`, no `Authorization` header. The files use the parquet schema underlying the row fields above (snake\_case column names: `ts` as epoch-millisecond INT64, `ua_raw`, `matched_operator`, `verdict_tier`, …) plus two archive-internal columns not present in `json`/`csv` rows: `stream_id` (ingest id) and `ip_asn` (org-level ASN of the source IP; `0` = unknown).
```sql theme={null}
-- DuckDB: load a whole export in one line, then query like a table.
SELECT matched_operator, count(*), sum(bytes)
FROM read_parquet(['', ''])
GROUP BY 1 ORDER BY 2 DESC;
```
**Treat parquet `url`s as secrets.** Anyone with a link can download that
day's full crawl log until `urlExpiresAt` — about **1 hour** — without
authenticating. Don't log these links or paste them anywhere public; request
fresh ones instead.
## Pagination
`json` and `csv` exports walk the archive oldest → newest and return `pageSize` rows per request. Pass the returned cursor (body `nextCursor`, or the `X-Next-Cursor` header for CSV) as `cursor` in the next request — **by itself**; the token pins everything else. Two properties worth knowing:
* Resuming is cheap: the cursor records an exact file position, so page 50 costs the same as page 2.
* Consistency is guaranteed: frozen archive files are write-once, and the cursor carries an integrity check — if the file behind your cursor changed you get `400 invalid_cursor` rather than silently shifted rows. Restart the export. This is rare for frozen days but *expected* for provisional days paginated across a day close (they're rewritten daily), so treat `invalid_cursor` as a normal restart signal in long-running exports.
### Error Responses
| Status | Body | Description |
| ------ | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 400 | `{ "error": "invalid_request", "message": "..." }` | Missing `domainId`; an invalid window, `format`, filter, or `pageSize`; a window over 31 days; filters combined with `format: "parquet"`; or a page that would fetch more archive data than one request may (narrow the window, add filters, or switch to `parquet`). The `message` field explains which. |
| 400 | `{ "error": "invalid_cursor", "message": "..." }` | The `cursor` token is malformed, or the underlying archive changed since it was minted. Restart from the first page. |
| 401 | `{ "error": "invalid_api_key" }` | API key is missing, invalid, or revoked. |
| 403 | `{ "error": "plan_upgrade_required" }` | Programmatic export requires the Pro plan or above. |
| 404 | `{ "error": "domain_not_found" }` | The `domainId` doesn't exist or isn't accessible to your account. |
| 429 | `{ "error": "rate_limit_exceeded_monthly", "usage": 300, "limit": 300 }` | Rate limit exceeded. The code is `rate_limit_exceeded_monthly` or `rate_limit_exceeded_per_minute` — match on the `rate_limit_exceeded` prefix to catch both. |
Errors return the JSON shapes above regardless of the requested `format`.
## Examples
### Which AI bots hit a section, verified only
```javascript theme={null}
async function verifiedAiHits(domainId) {
const rows = [];
let body = {
domainId,
period: "30d",
tier: "verified",
pathPrefix: "/docs/",
pageSize: 2000,
};
while (body) {
const res = await fetch("https://api.hadoseo.com/functions/v1/export-crawl-logs", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.HADOSEO_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (!res.ok) throw new Error(`${res.status}: ${(await res.json()).message}`);
const data = await res.json();
rows.push(...data.rows);
body = data.nextCursor ? { cursor: data.nextCursor } : null;
}
return rows;
}
```
### Bulk-loading history into DuckDB
```bash theme={null}
curl -s -X POST https://api.hadoseo.com/functions/v1/export-crawl-logs \
-H "Authorization: Bearer hado_sk_your_key_here" \
-H "Content-Type: application/json" \
-d '{"domainId": "11111111-1111-1111-1111-111111111111", "format": "parquet", "period": "30d"}' \
| python3 -c 'import json,sys; [print(f["url"]) for f in json.load(sys.stdin)["files"]]' \
| xargs -I{} curl -sO {}
```
Each API request counts against your monthly quota. For recurring pipelines,
prefer `format: "parquet"`: one request returns links to a full month of daily
files, versus potentially many paginated `json` requests for the same data.