# robots.txt — vocmap.com # # Three groups below, in order of what they do: # # 1. Search engines allowed everywhere, minus the raw data paths # 2. Google ad crawlers allowed everywhere, named explicitly # 3. AI / dataset crawlers disallowed everywhere # # A crawler obeys exactly ONE group — the most specific one that names it — so # every bot with its own block below ignores `User-agent: *` entirely. That is # why AdsBot-Google is repeated even though `*` already allows the site: Google # documents AdsBot as not following the wildcard group. # # This file is the machine-readable half of the "Automated access, data # extraction and AI training" section of https://vocmap.com/terms. The Terms are # what a court reads; this is what a crawler reads. Keep them saying the same # thing. # # The `Sitemap:` line is appended at build time by scripts/prerender.mjs from # VITE_SITE_URL — do not add it by hand here, or it will end up duplicated. # ── 1. Everything else, including Googlebot and Bingbot ───────────────────── # # Three data paths that USED to be disallowed are now allowed, because pages # render from them and blocking a render-critical file does not hide it — it # breaks the page for the crawler while leaving it visible to everyone else. # # /vocab-index/ VocabularyBrowsePage builds the per-type listings from # all.json. With this blocked, Googlebot rendered # /vocabulary/type/noun as the intro paragraph followed by # the literal text "0 entries" and an empty list — while a # human browser, which ignores robots.txt, saw all 30,942 # links. Twenty-three pages telling Google they were empty. # # /vocab-entries/ A word reached by clicking a family chip has no inlined # entry, so the page fetches its JSON. Blocked, that render # ends at "this word isn't in the dictionary yet". # # /demo/ The landing page's "Try it right here" section reads # demo-notes.json. Blocked, the homepage renders # "Demo conversation coming soon." to the one crawler whose # opinion of the homepage matters most. # # The old comment claimed "Nothing that matters for indexing is behind these # rules", and that was true only of a direct landing on a word page — the one # case where the entry is inlined. Everything else rendered short. # # None of these were protecting anything: the same data is already in the # served HTML, and robots.txt is a request to well-behaved crawlers, not an # access control. The Terms are what actually say no to scrapers. # # /notes/ stays blocked — it belongs to the signed-in study surface, nothing # public renders from it. /keywords/ likewise, and the path is corrected here: # the old rule said /keywords.json, which is not where the file lives, so it # matched nothing at all. User-agent: * Allow: / Disallow: /notes/ Disallow: /keywords/ # ── 2. Google advertising crawlers — must be named, must stay allowed ─────── # # Mediapartners-Google serves AdSense contextual targeting and AdsBot-Google # checks landing-page quality. Blocking either degrades ad serving, and the # AdSense review is open. Do not add Disallow lines to these two. User-agent: Mediapartners-Google Allow: / User-agent: AdsBot-Google Allow: / # ── 3. AI training and dataset crawlers — disallowed ──────────────────────── # # Google-Extended is the important one to understand: it opts the site out of # Gemini training WITHOUT affecting Search ranking or AdSense, because it is a # separate token from Googlebot. Blocking it costs nothing. # # Compliance here is voluntary. A crawler that ignores robots.txt is not # stopped by this file — but it has been given clear, machine-readable notice, # which is what turns "they took our data" into "they took our data after being # told not to." That distinction is the point. User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: meta-externalagent Disallow: / User-agent: FacebookBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: YouBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / # ── SEO tooling — crawls the whole site to resell the analysis ────────────── User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / Sitemap: https://vocmap.com/sitemap.xml