# Codext GmbH - robots.txt # https://codext.de # # Allow all standard search engines + explicit AI/LLM crawler allowlist # for AI Search Optimization (Google AI Overviews, ChatGPT, Perplexity, Claude, etc.) User-agent: * Allow: / Disallow: /private/ Disallow: /_codext/ Disallow: /api/ # --- Standard search engines ---------------------------------------------- User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Slurp Allow: / User-agent: Applebot Allow: / User-agent: YandexBot Allow: / User-agent: Baiduspider Allow: / # --- AI / LLM crawlers, explicitly allowed for AI Search Optimization ------- # robots.txt stays pure ASCII on purpose: it is the one file every crawler, # proxy and CDN cache in the chain touches, so it should not depend on any of # them honouring a charset. Prose with umlauts belongs in llms.txt. # OpenAI User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / # Anthropic User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / # Google AI User-agent: Google-Extended Allow: / User-agent: GoogleOther Allow: / # Apple AI User-agent: Applebot-Extended Allow: / # Perplexity User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Meta User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: FacebookBot Allow: / # Common Crawl (training data) User-agent: CCBot Allow: / # Mistral User-agent: MistralAI-User Allow: / # Cohere User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / # ByteDance / TikTok User-agent: Bytespider Allow: / # DuckDuckGo AI User-agent: DuckAssistBot Allow: / # You.com User-agent: YouBot Allow: / # Diffbot (used by AI products incl. Bard/Gemini training) User-agent: Diffbot Allow: / # Amazon User-agent: Amazonbot Allow: / # Sitemaps + LLM-friendly content map Sitemap: https://codext.de/sitemap-index.xml # # Plain-text / markdown mirrors maintained for AI agents: # /llms.txt - agency overview, services, facts, contact # /preise.md - pricing corridors per service, terms, project durations