Bot Definitions

POST /functions/v1/get-bot-definitions

Returns the bot taxonomy: every intent bucket (why a bot crawls) together with the full list of bots in it, who operates each one, and how its identity is verified. This is the decoder for the crawl exports. Export Crawl Analytics returns botTotals keyed by bot ID with an intent on each, and Export Page Analytics returns per-bot tier splits under pages[].bots — this endpoint tells you what those IDs and intents mean.

Always call this endpoint from a server-side environment (backend API, serverless function, build script, etc.). Never include your API key in client-side code — it will be visible to anyone inspecting your frontend.

Programmatic (API key) access requires the Pro plan and above — requests from Starter accounts return 403 plan_upgrade_required.

Cache on the version. The response is immutable for a given definitionsVersion. Read definitionsVersion from any export response you already fetch, and only call this endpoint when it reports a version you haven’t cached — typically once a month or less.

Versioning

The taxonomy is published as numbered, immutable snapshots starting at 1.0. Every response carries the version it came from and the date that version took effect.

Change	Bump	Example
A bot is added, removed, re-tagged, or its verification method changes	Minor (`1.0` → `1.1`)	A new AI crawler appears and is added to `training`
The set of intent buckets itself changes	Major (`1.4` → `2.0`)	A new bucket is introduced, or one is retired

A major bump is the signal that switching on intent in your own code may need updating. A minor bump never changes the set of possible intent values.

Request

Headers

Header	Required	Description
`Authorization`	Yes	`Bearer hado_sk_your_key_here`
`Content-Type`	Yes	`application/json`

Body Parameters

Parameter	Type	Required	Description
`version`	string	No	A specific published snapshot, e.g. `"1.0"`. Omit for the current one. Use this to decode an export you stored earlier: pass the `definitionsVersion` from that export’s response.

Example Request

Current taxonomy:

curl -X POST https://api.hadoseo.com/functions/v1/get-bot-definitions \
  -H "Authorization: Bearer hado_sk_your_key_here" \
  -H "Content-Type: application/json"

A specific historical snapshot:

curl -X POST https://api.hadoseo.com/functions/v1/get-bot-definitions \
  -H "Authorization: Bearer hado_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"version": "1.0"}'

Response

Success (200)

{
  "generatedAt": "2026-08-25T12:00:00.000Z",
  "definitionsVersion": "1.0",
  "definitionsEffectiveFrom": "2026-07-09T00:00:00.000Z",
  "isCurrent": true,
  "userId": "00000000-0000-0000-0000-000000000000",
  "intentCount": 8,
  "botCount": 94,
  "intents": [
    {
      "intent": "training",
      "label": "AI Training",
      "description": "Downloads your content to train AI models.",
      "isAiIntent": true,
      "botCount": 14
    },
    {
      "intent": "search-classic",
      "label": "Classic Search",
      "description": "A traditional search engine indexing your site.",
      "isAiIntent": false,
      "botCount": 14
    }
  ],
  "bots": [
    {
      "botId": "claudebot",
      "displayName": "ClaudeBot",
      "operator": "Anthropic",
      "intent": "training",
      "uaPatterns": ["claudebot"],
      "verificationMethod": "published_ip_ranges",
      "isActive": true
    },
    {
      "botId": "googlebot",
      "displayName": "Googlebot",
      "operator": "Google",
      "intent": "search-classic",
      "uaPatterns": ["googlebot"],
      "verificationMethod": "rdns_suffix",
      "isActive": true
    }
  ]
}

intents and bots are two flat arrays joined on intent — bots are not nested under their bucket. Looking a bot up by the ID you got from an export is a direct index into bots; listing a bucket’s members is one filter on intent.

Fields

Field	Type	Description
`generatedAt`	string	ISO 8601 timestamp of when the response was produced.
`definitionsVersion`	string	The taxonomy version this payload is, e.g. `1.0`. Immutable — safe to use as a cache key.
`definitionsEffectiveFrom`	string	ISO 8601 timestamp of when this version took effect.
`isCurrent`	boolean	`false` only when you explicitly requested an older `version`.
`userId`	string	The account that owns the API key.
`intentCount`	number	Number of intent buckets.
`botCount`	number	Number of bots in `bots`.
`intents[].intent`	string	The bucket’s identifier — the same value you see as `intent` in the crawl exports and on `bots[]` below.
`intents[].label`	string	Human-readable bucket name (e.g. `AI Training`).
`intents[].description`	string	One-sentence plain-English explanation.
`intents[].isAiIntent`	boolean	Whether this bucket counts as AI activity. See below.
`intents[].botCount`	number	Bots in this bucket, retired ones included. Filter on `isActive` for a “what can crawl me today” number.
`bots[].botId`	string	Stable bot identifier — the key used in `botTotals` and `pages[].bots`.
`bots[].displayName`	string	Human-readable bot name (e.g. `GPTBot`).
`bots[].operator`	string	The company running the bot (e.g. `OpenAI`). Group on this to roll several bots up into one vendor — most operators run more than one crawler, often across different intents (Google spans five).
`bots[].intent`	string	The bucket this bot belongs to — join key into `intents[]`.
`bots[].uaPatterns`	string[]	The lowercased substrings we match a User-Agent against to attribute a hit to this bot. See below.
`bots[].verificationMethod`	string	How this bot’s identity is confirmed. See below.
`bots[].isActive`	boolean	`false` for a bot we have retired from the taxonomy. See below.

Buckets are returned in display order (AI intents first); bots are sorted by botId.

Retired bots are included, not dropped. A bot we have stopped classifying stays in the payload with isActive: false, so decoding an archived export never hits an ID that resolves to nothing. For “which bots can crawl my site today”, filter bots on isActive.

AI intents. isAiIntent is true for exactly training, answer-fetch, and search-index. That is the definition behind “AI crawl volume” in the dashboard: models reading your content. search-classic (Googlebot, Bingbot) is deliberately excluded — including it would drown the AI signal in ordinary organic crawling.

Matching a User-Agent yourself

uaPatterns is the actual rule set we classify with, published so you can reproduce our attribution against your own access logs rather than taking the bot IDs on faith. They are lowercased substrings, not whole User-Agent strings and not regular expressions. To match a request the way we do:

Lowercase the request’s User-Agent.
Find every bot with a pattern contained in it.
The longest matching pattern wins. This matters: googlebot-image and googlebot are separate bots, and a Googlebot-Image request contains both. Matching shortest-first would collapse the whole Google suite into one bucket.
No match → the hit is counted under the unknown bot at the unknown tier. That entry is the synthetic catch-all, so its uaPatterns is deliberately empty — it is the bucket you fall into, never one you match.

function classify(userAgent, bots) {
  const ua = userAgent.toLowerCase();
  let best = null;
  for (const bot of bots) {
    for (const p of bot.uaPatterns) {
      if (ua.includes(p) && (!best || p.length > best.pattern.length)) {
        best = { bot, pattern: p };
      }
    }
  }
  return best?.bot ?? null;
}

A UA match alone only establishes the declared tier — anyone can send any User-Agent. Reaching verified additionally requires the source IP to check out, which is what verificationMethod describes below. If your own counts come out higher than ours, that difference is usually spoofed traffic we declined to attribute.

Verification methods

verificationMethod is how a bot can reach the verified attribution tier. A bot with none can never be verified, so it will only ever appear as declared in your exports — that is expected, not a data quality problem.

Method	Meaning
`rdns_suffix`	The source IP is reverse-DNS forward-confirmed against the operator’s hostname suffix (e.g. `.googlebot.com`).
`published_ip_ranges`	The source IP is checked against IP ranges the operator publishes (refreshed weekly).
`asn_only`	Verified at the network-operator level only.
`none`	The operator publishes nothing verifiable. Traffic from this bot is always `declared`.

Error Responses

Status	Body	Description
400	`{ "error": "invalid_request" }`	`version` was malformed, or names a version that doesn’t exist
401	`{ "error": "invalid_api_key" }`	API key is missing, invalid, or revoked
403	`{ "error": "plan_upgrade_required" }`	Programmatic access requires the Pro plan or above
429	`{ "error": "rate_limit_exceeded", "usage": 300, "limit": 300 }`	Monthly or per-minute rate limit exceeded

Examples

JavaScript / Node.js

The version-aware pattern — fetch definitions only when the version changes, then decode an export’s bot keys into display names:

const HEADERS = {
  "Authorization": `Bearer ${process.env.HADOSEO_API_KEY}`,
  "Content-Type": "application/json",
};

const cache = new Map(); // definitionsVersion -> { byBotId, byIntent }

async function getDefinitions(version) {
  if (cache.has(version)) return cache.get(version);
  const res = await fetch(
    "https://api.hadoseo.com/functions/v1/get-bot-definitions",
    { method: "POST", headers: HEADERS, body: JSON.stringify({ version }) },
  );
  const { intents, bots } = await res.json();
  const defs = {
    byBotId: new Map(bots.map((b) => [b.botId, b])),
    byIntent: new Map(intents.map((i) => [i.intent, i])),
  };
  cache.set(version, defs);
  return defs;
}

// 1. Pull a crawl export. Its envelope tells you which taxonomy produced it.
const exportRes = await fetch(
  "https://api.hadoseo.com/functions/v1/export-crawl-analytics",
  {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({ domainId: "your-domain-id", period: "30d" }),
  },
);
const report = await exportRes.json();

// 2. Fetch that exact taxonomy version (cached after the first call).
const { byBotId, byIntent } = await getDefinitions(report.definitionsVersion);

// 3. Decode the export's bot keys and report AI crawl volume.
const [{ botTotals }] = report.domains;
for (const [botId, totals] of Object.entries(botTotals)) {
  const bot = byBotId.get(botId);
  if (!bot || !byIntent.get(bot.intent)?.isAiIntent) continue;
  console.log(`${bot.displayName} (${bot.operator}): ${totals.total} crawls`);
}

A handful of bot keys on days before the legacy cutover are aggregates with no taxonomy entry and carry "intent": null. Skip keys that don’t resolve, as the example above does.

Overview

Endpoints

Versioning

Request

Headers

Body Parameters

Example Request

Response

Success (200)

Fields

Matching a User-Agent yourself

Verification methods

Error Responses

Examples

JavaScript / Node.js

​Versioning

​Request

​Headers

​Body Parameters

​Example Request

​Response

​Success (200)

​Fields

​Matching a User-Agent yourself

​Verification methods

​Error Responses

​Examples

​JavaScript / Node.js

Versioning

Request

Headers

Body Parameters

Example Request

Response

Success (200)

Fields

Matching a User-Agent yourself

Verification methods

Error Responses

Examples

JavaScript / Node.js