llms.txt — AI & LLM Crawler Rules
Version: 1.0
This file controls how AI agents, LLMs, and dataset collectors may access or use site content.
------------------------------
DEFAULT: Allow AI Search & Preview
------------------------------
User-Agent: *
Allow: /
AI-Search: allow
AI-Preview: allow
AI-Snippet: allow

------------------------------
ALLOW MAJOR AI SEARCH CRAWLERS
------------------------------
OpenAI ChatGPT / GPT Search
User-Agent: GPTBot
Allow: /

OpenAI "ChatGPT-Search" (2024+)
User-Agent: ChatGPT-User
Allow: /

Google Gemini / Bard / Search-GenAI
User-Agent: Google-Extended
Allow: /

Google Experimental Crawler
User-Agent: GoogleOther
Allow: /

Anthropic Claude Crawler
User-Agent: Claude-Web
Allow: /

Meta (Facebook) AI
User-Agent: FacebookAI
Allow: /

Perplexity AI
User-Agent: PerplexityBot
Allow: /

xAI (Grok)
User-Agent: xAI
Allow: /
User-Agent: GrokCrawler
Allow: /

Apple AI Search
User-Agent: Applebot-Extended
Allow: /

Amazon AI (Alexa/Titan)
User-Agent: Amazonbot
Allow: /

Mistral AI
User-Agent: MistralAI
Allow: /

HuggingFace Research Crawler
User-Agent: HuggingFace
Allow: /

Chinese Ecosystem (optional)
User-Agent: Baidu
Allow: /
User-Agent: Alibaba-AI
Allow: /
User-Agent: Tencent-AI
Allow: /
User-Agent: ByteDance-AI
Allow: /

------------------------------
CONTACT
------------------------------
Contact: info@devicesfinder.com