User-agent: * Allow: / # Disallow paths Disallow: /api/ Disallow: /_next/ Disallow: /admin/ # Note: no `Disallow: /*.json$` rule. /openapi.json is the ONLY .json URL this # site serves and it is the API discovery document we want fetched. A # `$`-anchored wildcard is more specific than `Allow: /`, so such a rule wins # on longest-match and silently blocked it. Next.js data JSON is already # covered by the /_next/ rule above. # Host Host: https://www.fiskil.com # Sitemaps Sitemap: https://www.fiskil.com/sitemap-index.xml # AI discoverability # https://llmstxt.org/ LLMs-Txt: https://www.fiskil.com/llms.txt LLMs-Full-Txt: https://www.fiskil.com/llms-full.txt # Search & citation crawlers. # These fetch pages to answer questions and to attribute sources, so they are # explicitly allowed. A per-agent group does not inherit the rules above, so the # disallow list is repeated here. # Note: OAI-SearchBot (not GPTBot) is the agent behind ChatGPT search citations. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Applebot Allow: / Disallow: /api/ Disallow: /_next/ Disallow: /admin/ # Model-training crawlers (GPTBot, CCBot, ClaudeBot, Google-Extended, # Applebot-Extended, Bytespider, Amazonbot, Meta-ExternalAgent, cohere-ai) are # intentionally not listed and therefore follow the "User-agent: *" group above. # Restricting them is a separate policy decision and does not affect the # citation crawlers listed above.