# Open to search as of 2026-08-16 (go-live signed off by Sachith).
#
# This replaced a full pre-launch lock (`Disallow: /`). If the site ever needs pulling
# back out of search, restoring that lock is only ONE of three layers — see
# docs/go-live-checklist.md §0. The others are `public/_headers` (deleted at launch)
# and PUBLIC_ALLOW_INDEX (src/lib/noindex.ts). All three must move together.
#
# Do NOT delete this file to "open" the site — an absent robots.txt loses the /api/
# disallow and both Sitemap declarations, which is how search engines discover the
# URL inventory.
#
# NOTE: robots.txt is advisory — honoured by legitimate search engines (Google/Bing),
# ignored by bad-actor scrapers. For a hard wall, add a Cloudflare WAF/Access rule.

# NOTE ON RULE ORDER: `Disallow: /api/` is listed BEFORE `Allow: /` in every group
# on purpose. RFC 9309 crawlers (Google, Bing, the AI operators below) apply the
# LONGEST match and get this right either way, but simpler crawlers apply the FIRST
# match in file order and would read a leading `Allow: /` as permission for /api/.
# This order is correct under both models. Keep it.

User-agent: *
Disallow: /api/
Allow: /

# ---------------------------------------------------------------------------
# AI crawlers — MIRRORS THE CLOUDFLARE EDGE RULE (decision: Sachith, 2026-08-18).
#
# Policy: block bulk training/scraping, allow every live retrieval fetcher.
#
# WHY THIS SHAPE. Cloudflare's "Block AI Scrapers and Crawlers" rule is ON for
# this zone and returns 403 at the edge, before a request ever reaches the
# Worker or this file. Verified 2026-08-18 by user-agent probe against
# /intelligence/rare-hire-record/:
#     403 → GPTBot · CCBot · meta-externalagent · Bytespider · Amazonbot
#     200 → ClaudeBot · Google-Extended · Applebot-Extended
#     200 → ChatGPT-User · OAI-SearchBot · Claude-User · Claude-SearchBot
#           PerplexityBot · Perplexity-User
# The groups below restate that same policy so the file does not advertise
# access the edge refuses. THE EDGE IS THE ENFORCER; this file is the statement
# of intent. If you change one, change the other, or they drift apart again.
#
# The retrieval fetchers are all allowed on purpose — they fetch a page at the
# moment a user asks a question, so they are what produces CITATIONS. Blocking
# them would remove us from AI answers, the opposite of the GEO/AEO goal (be the
# cited source on GCC hiring data). Training crawlers get us nothing measurable
# in return, which is why they are the half that is refused.
#
# NOTE: the edge rule is not a clean training-vs-retrieval split — it blocks
# GPTBot but allows ClaudeBot, Google-Extended and Applebot-Extended, which are
# also training crawlers. That is Cloudflare's list, not our reasoning. We mirror
# it rather than invent a third policy. If you want those three blocked too,
# change them here AND add a WAF rule, since robots.txt alone is advisory.
#
# HOW ROBOTS.TXT MATCHING WORKS — the thing that bites people:
#   A crawler obeys exactly ONE group, the most specific match on its name. Once
#   a bot is named below it STOPS reading the `*` group entirely, so every rule
#   that must still apply to it has to be repeated inside its own group. That is
#   why `Disallow: /api/` appears in each allowed block rather than being
#   inherited. A `Disallow: /` group needs nothing further — it already covers
#   /api/.
#
# Our proprietary datasets (Rare-Hire Record detail, gated reports) sit behind
# capture forms, so crawlers only ever reach the public marketing layer.
#
# To reverse for a single provider, flip its rule — do NOT delete the group, or
# it silently falls back to `*` and becomes allowed.
#
# robots.txt is advisory: honoured by these operators, ignored by bad-actor
# scrapers. Note the edge rule matches the user-agent as a CASE-SENSITIVE
# SUBSTRING, so `gptbot` lowercase is not caught by it. Neither layer stops
# anyone who lies about who they are.
# ---------------------------------------------------------------------------

# --- Bulk training / scraping crawlers: blocked (matches the edge) ---

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /api/
Allow: /

User-agent: Google-Extended
Disallow: /api/
Allow: /

User-agent: Applebot-Extended
Disallow: /api/
Allow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

# --- Live retrieval fetchers: allowed. These generate the citations. ---

User-agent: ChatGPT-User
Disallow: /api/
Allow: /

User-agent: OAI-SearchBot
Disallow: /api/
Allow: /

User-agent: Claude-User
Disallow: /api/
Allow: /

User-agent: Claude-SearchBot
Disallow: /api/
Allow: /

User-agent: PerplexityBot
Disallow: /api/
Allow: /

User-agent: Perplexity-User
Disallow: /api/
Allow: /

Sitemap: https://recruiseglobal.com/sitemap-index.xml
Sitemap: https://recruiseglobal.com/sitemap-jobs.xml