AI Bot & llms.txt Access Scanner
Evaluates AI indexing posture. Scans firewall blocking status for 25+ major AI crawlers, parses token-efficient llms.txt files according to the standard, and checks whether your site is accessible or walled off from LLM retrieval engines.
This precision diagnostic tool audits diagnostic instrument against international technical standards. It verifies structural integrity, uncovers hidden security flaws, and outputs deterministic diagnostic reports entirely within browser memory or through zero-retention edge lookups.
Diagnostic Inspection Specifications
| What We Scan & Verify | Technical Standard / Threshold |
|---|---|
| Binary / Protocol Conformance | International Open Standard Specification |
| Cryptographic Integrity Digest | SHA-256 (FIPS PUB 180-4) |
| Privacy Leakage & Telemetry | Zero Cloud Retention (100% In-Memory) |
| Diagnostic Latency Benchmark | Sub-200ms Execution Target |
Architecture Comparison: OnChainScans vs Legacy Web Checkers
| Diagnostic Capability | OnChainScans (Zero-Telemetry) | Traditional Web Scanners |
|---|---|---|
| File Execution Boundary | 100% Client-Side WebAssembly | Uploaded to third-party server |
| Data Retention & Logs | Zero Logs (Discarded upon tab close) | Stored in database / analytics |
| Execution Engine | Live RFC / Public APIs / Raw Bytes | Simulated / heuristic estimates |
| Subscription / Paywall | 100% Free Forever (Open Tool) | Sign-up gating or credit limits |
How This Scan Works Under the Hood
The scanner fetches /robots.txt and /llms.txt directly from the target domain, evaluating crawler authorization rules across 10 major AI bots.
It audits user-agents including GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Bytespider, determining whether each is explicitly allowed, blocked, or governed by wildcard '*' directives.
It parses /llms.txt and /llms-full.txt against the emerging 2026 standard, checking for Markdown H1 titles, summaries, and token-efficient documentation indexes.
Frequently Asked Questions
What is the /llms.txt standard?
/llms.txt is a proposed standard similar to robots.txt that provides condensed, structured Markdown context for Large Language Models to read without parsing complex web layouts.
Does blocking GPTBot stop ChatGPT from showing my site in search?
Blocking GPTBot stops OpenAI from using your site for model training. ChatGPT live web browsing uses a separate agent ('ChatGPT-User').
What is Google-Extended?
Google-Extended is the user-agent Google uses to train Gemini and Vertex AI models. Blocking it does not affect normal Google Search ranking.
What technical standards does this tool enforce?
This tool strictly validates structures against official IETF RFCs, ISO standards, NIST guidelines, and W3C specifications without relying on simulated or mock heuristics.
Related Web, SEO & Network Chains Instruments
View all in categorySSL Certificate Chain Scanner
Dissects the complete X.509 certificate chain from leaf to root authority. Evaluates OCSP stapling, Certificate Transparency (CT) logs, and weak cipher suites.
Redirect & Canonical Chain Scanner
Follows every HTTP hop across 301, 302, 307, 308, and JavaScript meta-refreshes. Correlates HTTP status codes with rel='canonical' declarations to detect SEO leakage.
Dangling CNAME & Subdomain Takeover Scanner
Scans DNS CNAME records for vulnerable pointers to decommissioned AWS S3, GitHub Pages, Vercel, Heroku, or Azure endpoints vulnerable to malicious takeover.