# Fridge.com Robots.txt
# Last updated: December 27, 2025
# Site is LIVE and indexed

# ==============================================
# UNIVERSAL DISALLOW RULES (Apply to ALL bots)
# ==============================================
# Note: In robots.txt, each User-agent block is independent.
# We must repeat Disallow rules in each block, OR use User-agent: * first.
# We use User-agent: * with Disallows, then specific bots inherit by matching *.

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin-panel/
Disallow: /_next/
Disallow: /auth/
Disallow: /signin
Disallow: /signup
Disallow: /docs/
Disallow: /r/

# ==============================================
# AI CRAWLERS — EXPLICITLY WELCOME
# (All 7 Major LLMs + More)
# ==============================================
# These bots match User-agent: * above, which allows / but blocks sensitive paths.
# The comments below document which AI crawlers we welcome.

# OpenAI (ChatGPT): GPTBot, ChatGPT-User
# Google AI (Gemini): Google-Extended
# Anthropic (Claude): ClaudeBot, Claude-Web
# Perplexity: PerplexityBot
# Microsoft (Copilot): Bingbot
# Meta (Llama): Meta-ExternalAgent, FacebookBot
# xAI (Grok): (uses standard web crawling)
# Common Crawl: CCBot
# Cohere: cohere-ai
# You.com: YouBot

# ==============================================
# SEARCH ENGINE CRAWLERS — WELCOME
# ==============================================
# Googlebot, Bingbot, DuckDuckBot, Slurp, Yandex, Applebot
# All match User-agent: * above

# ==============================================
# SITEMAPS
# ==============================================

Sitemap: https://fridge.com/sitemap.xml

# ==============================================
# LLM-SPECIFIC FILES
# ==============================================

# For AI assistants: comprehensive site information
# ChatGPT, Perplexity, Gemini, Copilot, Claude, Llama, Grok, and more
# See: https://fridge.com/llms.txt