AI Robots Policy Check
Read robots.txt and report homepage access for GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended and others.
https://milliapi.com/api/ai-robots-check?url=https%3A%2F%2Fexample.comMilliAPI sells deterministic web utilities directly to AI agents through x402. Services start at one tenth of a cent. Discover an endpoint, pay USDC on Base, and get JSON back.
GET /api/ai-robots-check?url=https://example.com 402 Payment Required price: $0.001 USDC network: Base protocol: x402 v2 agent pays automatically ↓ 200 OK { "GPTBot": { "allowed": true } }
One input: a public HTTPS URL. No buyer account or API key.
Read robots.txt and report homepage access for GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended and others.
https://milliapi.com/api/ai-robots-check?url=https%3A%2F%2Fexample.comCheck whether a site publishes llms.txt and return the HTTP status, byte size and a bounded content preview.
https://milliapi.com/api/llms-txt-check?url=https%3A%2F%2Fexample.comExtract title, description, canonical URL, robots meta, Open Graph fields, H1 count and JSON-LD count from a public page.
https://milliapi.com/api/page-metadata?url=https%3A%2F%2Fexample.comCombine crawler policy, robots.txt, llms.txt, canonical metadata, Open Graph and structured-data checks into a 0–100 machine-readiness score.
https://milliapi.com/api/agent-web-audit?url=https%3A%2F%2Fexample.comDiscovery and payment are machine-readable.
Bazaar metadata, OpenAPI, llms.txt and a free catalog describe each service.
x402 v2 requests USDC on Base. Compatible clients authorize micropayments programmatically.
Settlement events and live practice checks feed a bounded graph so discovery, documentation and reliability can improve without touching wallet or settlement controls.
Free discovery and optimization endpoints.