> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hadoseo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Bot Definitions

> The published bot taxonomy — every intent bucket and the bots in it, versioned

```
POST /functions/v1/get-bot-definitions
```

Returns the **bot taxonomy**: every intent bucket (why a bot crawls) together with the full list of bots in it, who operates each one, and how its identity is verified.

This is the decoder for the crawl exports. [Export Crawl Analytics](/api-reference/endpoint/export-crawl-analytics) returns `botTotals` keyed by bot ID with an `intent` on each, and [Export Page Analytics](/api-reference/endpoint/export-page-analytics) returns per-bot tier splits under `pages[].bots` — this endpoint tells you what those IDs and intents mean.

<Warning>
  Always call this endpoint from a **server-side environment** (backend API, serverless function, build script, etc.). Never include your API key in client-side code — it will be visible to anyone inspecting your frontend.
</Warning>

<Info>
  Programmatic (API key) access requires the **Pro plan and above** — requests
  from Starter accounts return `403 plan_upgrade_required`.
</Info>

<Tip>
  **Cache on the version.** The response is immutable for a given `definitionsVersion`. Read `definitionsVersion` from any export response you already fetch, and only call this endpoint when it reports a version you haven't cached — typically once a month or less.
</Tip>

## Versioning

The taxonomy is published as numbered, immutable snapshots starting at `1.0`. Every response carries the version it came from and the date that version took effect.

| Change                                                                 | Bump                      | Example                                             |
| ---------------------------------------------------------------------- | ------------------------- | --------------------------------------------------- |
| A bot is added, removed, re-tagged, or its verification method changes | **Minor** (`1.0` → `1.1`) | A new AI crawler appears and is added to `training` |
| The **set of intent buckets** itself changes                           | **Major** (`1.4` → `2.0`) | A new bucket is introduced, or one is retired       |

A major bump is the signal that switching on `intent` in your own code may need updating. A minor bump never changes the set of possible `intent` values.

## Request

### Headers

| Header          | Required | Description                    |
| --------------- | -------- | ------------------------------ |
| `Authorization` | Yes      | `Bearer hado_sk_your_key_here` |
| `Content-Type`  | Yes      | `application/json`             |

### Body Parameters

| Parameter | Type   | Required | Description                                                                                                                                                                        |
| --------- | ------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `version` | string | No       | A specific published snapshot, e.g. `"1.0"`. Omit for the current one. Use this to decode an export you stored earlier: pass the `definitionsVersion` from that export's response. |

### Example Request

Current taxonomy:

```bash theme={null}
curl -X POST https://api.hadoseo.com/functions/v1/get-bot-definitions \
  -H "Authorization: Bearer hado_sk_your_key_here" \
  -H "Content-Type: application/json"
```

A specific historical snapshot:

```bash theme={null}
curl -X POST https://api.hadoseo.com/functions/v1/get-bot-definitions \
  -H "Authorization: Bearer hado_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"version": "1.0"}'
```

## Response

### Success (200)

```json theme={null}
{
  "generatedAt": "2026-08-25T12:00:00.000Z",
  "definitionsVersion": "1.0",
  "definitionsEffectiveFrom": "2026-07-09T00:00:00.000Z",
  "isCurrent": true,
  "userId": "00000000-0000-0000-0000-000000000000",
  "intentCount": 8,
  "botCount": 94,
  "intents": [
    {
      "intent": "training",
      "label": "AI Training",
      "description": "Downloads your content to train AI models.",
      "isAiIntent": true,
      "botCount": 14
    },
    {
      "intent": "search-classic",
      "label": "Classic Search",
      "description": "A traditional search engine indexing your site.",
      "isAiIntent": false,
      "botCount": 14
    }
  ],
  "bots": [
    {
      "botId": "claudebot",
      "displayName": "ClaudeBot",
      "operator": "Anthropic",
      "intent": "training",
      "uaPatterns": ["claudebot"],
      "verificationMethod": "published_ip_ranges",
      "isActive": true
    },
    {
      "botId": "googlebot",
      "displayName": "Googlebot",
      "operator": "Google",
      "intent": "search-classic",
      "uaPatterns": ["googlebot"],
      "verificationMethod": "rdns_suffix",
      "isActive": true
    }
  ]
}
```

`intents` and `bots` are two flat arrays joined on `intent` — bots are not nested under their bucket. Looking a bot up by the ID you got from an export is a direct index into `bots`; listing a bucket's members is one filter on `intent`.

### Fields

| Field                       | Type      | Description                                                                                                                                                                                        |
| --------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `generatedAt`               | string    | ISO 8601 timestamp of when the response was produced.                                                                                                                                              |
| `definitionsVersion`        | string    | The taxonomy version this payload is, e.g. `1.0`. Immutable — safe to use as a cache key.                                                                                                          |
| `definitionsEffectiveFrom`  | string    | ISO 8601 timestamp of when this version took effect.                                                                                                                                               |
| `isCurrent`                 | boolean   | `false` only when you explicitly requested an older `version`.                                                                                                                                     |
| `userId`                    | string    | The account that owns the API key.                                                                                                                                                                 |
| `intentCount`               | number    | Number of intent buckets.                                                                                                                                                                          |
| `botCount`                  | number    | Number of bots in `bots`.                                                                                                                                                                          |
| `intents[].intent`          | string    | The bucket's identifier — the same value you see as `intent` in the crawl exports and on `bots[]` below.                                                                                           |
| `intents[].label`           | string    | Human-readable bucket name (e.g. `AI Training`).                                                                                                                                                   |
| `intents[].description`     | string    | One-sentence plain-English explanation.                                                                                                                                                            |
| `intents[].isAiIntent`      | boolean   | Whether this bucket counts as AI activity. See below.                                                                                                                                              |
| `intents[].botCount`        | number    | Bots in this bucket, retired ones included. Filter on `isActive` for a "what can crawl me today" number.                                                                                           |
| `bots[].botId`              | string    | Stable bot identifier — the key used in `botTotals` and `pages[].bots`.                                                                                                                            |
| `bots[].displayName`        | string    | Human-readable bot name (e.g. `GPTBot`).                                                                                                                                                           |
| `bots[].operator`           | string    | The company running the bot (e.g. `OpenAI`). Group on this to roll several bots up into one vendor — most operators run more than one crawler, often across different intents (Google spans five). |
| `bots[].intent`             | string    | The bucket this bot belongs to — join key into `intents[]`.                                                                                                                                        |
| `bots[].uaPatterns`         | string\[] | The lowercased substrings we match a User-Agent against to attribute a hit to this bot. See below.                                                                                                 |
| `bots[].verificationMethod` | string    | How this bot's identity is confirmed. See below.                                                                                                                                                   |
| `bots[].isActive`           | boolean   | `false` for a bot we have retired from the taxonomy. See below.                                                                                                                                    |

Buckets are returned in display order (AI intents first); bots are sorted by `botId`.

<Note>
  **Retired bots are included, not dropped.** A bot we have stopped classifying stays in the payload with `isActive: false`, so decoding an archived export never hits an ID that resolves to nothing. For "which bots can crawl my site today", filter `bots` on `isActive`.
</Note>

<Note>
  **AI intents.** `isAiIntent` is `true` for exactly `training`, `answer-fetch`, and `search-index`. That is the definition behind "AI crawl volume" in the dashboard: models reading your content. `search-classic` (Googlebot, Bingbot) is deliberately excluded — including it would drown the AI signal in ordinary organic crawling.
</Note>

### Matching a User-Agent yourself

`uaPatterns` is the actual rule set we classify with, published so you can reproduce our attribution against your own access logs rather than taking the bot IDs on faith.

They are **lowercased substrings**, not whole User-Agent strings and not regular expressions. To match a request the way we do:

1. Lowercase the request's `User-Agent`.
2. Find every bot with a pattern contained in it.
3. **The longest matching pattern wins.** This matters: `googlebot-image` and `googlebot` are separate bots, and a Googlebot-Image request contains both. Matching shortest-first would collapse the whole Google suite into one bucket.
4. No match → the hit is counted under the `unknown` bot at the `unknown` tier. That entry is the synthetic catch-all, so its `uaPatterns` is deliberately empty — it is the bucket you fall into, never one you match.

```javascript theme={null}
function classify(userAgent, bots) {
  const ua = userAgent.toLowerCase();
  let best = null;
  for (const bot of bots) {
    for (const p of bot.uaPatterns) {
      if (ua.includes(p) && (!best || p.length > best.pattern.length)) {
        best = { bot, pattern: p };
      }
    }
  }
  return best?.bot ?? null;
}
```

<Note>
  A UA match alone only establishes the `declared` tier — anyone can send any User-Agent. Reaching `verified` additionally requires the source IP to check out, which is what `verificationMethod` describes below. If your own counts come out higher than ours, that difference is usually spoofed traffic we declined to attribute.
</Note>

### Verification methods

`verificationMethod` is **how** a bot can reach the `verified` [attribution tier](/api-reference/endpoint/export-crawl-analytics#attribution-tiers). A bot with `none` can never be verified, so it will only ever appear as `declared` in your exports — that is expected, not a data quality problem.

| Method                | Meaning                                                                                                        |
| --------------------- | -------------------------------------------------------------------------------------------------------------- |
| `rdns_suffix`         | The source IP is reverse-DNS forward-confirmed against the operator's hostname suffix (e.g. `.googlebot.com`). |
| `published_ip_ranges` | The source IP is checked against IP ranges the operator publishes (refreshed weekly).                          |
| `asn_only`            | Verified at the network-operator level only.                                                                   |
| `none`                | The operator publishes nothing verifiable. Traffic from this bot is always `declared`.                         |

### Error Responses

| Status | Body                                                             | Description                                                    |
| ------ | ---------------------------------------------------------------- | -------------------------------------------------------------- |
| 400    | `{ "error": "invalid_request" }`                                 | `version` was malformed, or names a version that doesn't exist |
| 401    | `{ "error": "invalid_api_key" }`                                 | API key is missing, invalid, or revoked                        |
| 403    | `{ "error": "plan_upgrade_required" }`                           | Programmatic access requires the Pro plan or above             |
| 429    | `{ "error": "rate_limit_exceeded", "usage": 300, "limit": 300 }` | Monthly or per-minute rate limit exceeded                      |

## Examples

### JavaScript / Node.js

The version-aware pattern — fetch definitions only when the version changes, then decode an export's bot keys into display names:

```javascript theme={null}
const HEADERS = {
  "Authorization": `Bearer ${process.env.HADOSEO_API_KEY}`,
  "Content-Type": "application/json",
};

const cache = new Map(); // definitionsVersion -> { byBotId, byIntent }

async function getDefinitions(version) {
  if (cache.has(version)) return cache.get(version);
  const res = await fetch(
    "https://api.hadoseo.com/functions/v1/get-bot-definitions",
    { method: "POST", headers: HEADERS, body: JSON.stringify({ version }) },
  );
  const { intents, bots } = await res.json();
  const defs = {
    byBotId: new Map(bots.map((b) => [b.botId, b])),
    byIntent: new Map(intents.map((i) => [i.intent, i])),
  };
  cache.set(version, defs);
  return defs;
}

// 1. Pull a crawl export. Its envelope tells you which taxonomy produced it.
const exportRes = await fetch(
  "https://api.hadoseo.com/functions/v1/export-crawl-analytics",
  {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({ domainId: "your-domain-id", period: "30d" }),
  },
);
const report = await exportRes.json();

// 2. Fetch that exact taxonomy version (cached after the first call).
const { byBotId, byIntent } = await getDefinitions(report.definitionsVersion);

// 3. Decode the export's bot keys and report AI crawl volume.
const [{ botTotals }] = report.domains;
for (const [botId, totals] of Object.entries(botTotals)) {
  const bot = byBotId.get(botId);
  if (!bot || !byIntent.get(bot.intent)?.isAiIntent) continue;
  console.log(`${bot.displayName} (${bot.operator}): ${totals.total} crawls`);
}
```

<Note>
  A handful of bot keys on days before the [legacy cutover](/api-reference/endpoint/export-crawl-analytics#legacy-data-before-the-cutover) are aggregates with no taxonomy entry and carry `"intent": null`. Skip keys that don't resolve, as the example above does.
</Note>