Docs & FAQ

Everything about how the scanner works, what it checks, and how to use the output.

🤖

1. AI Index & Audit

Crawls your key pages respecting robots.txt (max 25 pages per free scan), converts them to clean Markdown, and scores AI-readiness 0–100 across five weighted categories: Crawlability (25), Structure (20), Machine-readable metadata (20), LLM-readiness (15+10), Content density (10). Every point lost comes with a finding and a fix.

📝

2. /llms.txt & /llms-full.txt generation

From the crawl we generate the emerging community standard: a concise /llms.txt index (H1, blockquote summary, Docs + Optional sections) and /llms-full.txt (full markdown dump, one source-labeled section per page). Download both and upload them to your web root — AI agents like GPTBot, ClaudeBot and PerplexityBot read them directly.

🔓

3. Markdown Agent Proxy

Every scanned site is available at /proxy/{domain}/ — clean markdown per page with Content-Type: text/markdown, cache headers and a Last-Crawled header. Point AI agents and RAG pipelines at it instead of your JS-heavy HTML. Rate-limited and served straight from disk cache.

🔒

4. Security & Headers

Grades CSP, HSTS, X-Content-Type-Options, clickjacking protection (X-Frame-Options / frame-ancestors), Referrer-Policy, Permissions-Policy and COOP. Also checks TLS certificate expiry via live handshake, Secure/HttpOnly/SameSite cookie flags, server-version disclosure and mixed content.

🌐

5. DNS & Network

Full record set (A, AAAA, CNAME, MX, NS, TXT, CAA, SOA) via DNS-over-HTTPS, nameserver redundancy, email-spoofing protection (SPF/DMARC with strength analysis, DKIM selector probe), CAA policy, and WHOIS facts (registrar, created/expires, registry status) via the IANA RDAP bootstrap.

⚡

6. Speed & Vitals

Timed TTFB, HTML weight, gzip/brotli compression, redirect-chain latency, asset HEAD probes (count, bytes, caching headers, compression), render-blocking head scripts, and image hygiene (lazy-loading, dimensions, alt text).

🐞

7. Bugs & Glitches

Broken internal links (batched HEAD→GET with concurrency cap), redirecting links, duplicate titles, missing meta descriptions, missing canonicals, multiple/missing H1s, missing alt text and charset issues. Site-wide checks activate automatically when the AI crawl runs in the same scan.

❓

FAQ: Is anything installed on my site?

No. The scanner only fetches public URLs like any visitor or crawler would. Uploading the generated llms.txt is optional (but recommended).

❓

FAQ: My site is a React/Vue SPA — why are pages empty?

This tool uses static HTML fetching (no headless browser) on purpose: it is fast and runs on modest hosting. If pages come back JS-only, that is itself a finding — AI agents that don't render JavaScript see the same emptiness. Serve SSR/static versions of key pages.

❓

FAQ: Where do the /llms.txt files go?

Upload them to your web root so they serve at https://yourdomain.com/llms.txt and /llms-full.txt. Or keep using the Markdown Agent Proxy — no upload needed.

❓

FAQ: Which AI bots do you check in robots.txt?

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, Bytespider and meta-externalagent — plus a generic check that your own crawler user-agent is allowed.

🛠

API reference

POST /api/scan {"domain":"example.com","modules":["ai","security","network","speed","faults"]} → full report JSON
GET /api/report/:siteId → cached report · GET /api/report/:siteId/report.md → markdown report
GET /api/report/:siteId/llms.txt · /llms-full.txt · /pages → generated artifacts
GET /proxy/:domain/ → cached markdown per page (rate-limited)
GET /api/health → liveness