Audit robots.txt access for AI crawlers
One broad disallow, an unexpected wildcard, or a host mismatch can keep AI search and agent crawlers away from otherwise public content. MilliAPI evaluates the effective policy for named AI bots.
No account, subscription, or API key is required. Paid calls use x402 with USDC on Base: request the endpoint, inspect the HTTP 402 terms, authorize payment under your own wallet policy, then retry with the payment signature.
Who it is for
Useful for technical SEO teams, publishers, security reviewers, and developers who need intentional—not accidental—AI crawler access.
What the paid result contains
The paid response reports each evaluated crawler, its access decision, the relevant policy evidence, and the checked timestamp without changing the target site.
What MilliAPI checks
- robots.txt retrieval and final URL
- effective allow or disallow decisions for major AI crawlers
- missing or conflicting policy signals
- structured evidence suitable for CI or monitoring
Representative result
Fields vary with the target and live evidence; this abbreviated example shows the response shape, not a promised score.
{"robots":{"exists":true,"status":200},"crawlers":{"OAI-SearchBot":"allowed","GPTBot":"blocked"}}Current price: $0.001 per successful response, derived from MilliAPI’s product manifest. Unqualified or invalid work is not presented as a successful paid result.