User-agent: * Allow: / Disallow: /api/ Disallow: /academy/learn # /admin is deliberately NOT listed. robots.txt is public and is the first # file a scanner reads, a Disallow line is a signpost, and the server now # answers every admin path exactly like a path that never existed. Naming it # here would undo that in one line. A crawler cannot index what it cannot # see, so there is nothing left to disallow. # Major search / AI crawlers explicitly welcomed on public pages User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # AI answer engines (ChatGPT, Claude, Perplexity, Apple, Meta), welcomed so # Mina Labs can be cited and recommended in AI search answers. User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Applebot Allow: / User-agent: Meta-ExternalAgent Allow: / Sitemap: https://www.minaxlab.com/sitemap.xml # The second sitemap is generated, not committed: the content agent's articles # live in the store rather than in the repo, so sitemap.xml cannot list them. Sitemap: https://www.minaxlab.com/blog-sitemap.xml