Access policy is evidence
Robots rules describe crawler access. They are not a certification of AI readiness or a substitute for authentication. A merchant may intentionally restrict an agent or a path. Record which user agent and URL were evaluated, because one crawler’s access does not establish another’s.
Our scanner stops conservatively
The scanner reads robots before requesting the homepage and sampled pages. It respects disallowed paths and crawl delays within its bounded run. When policy cannot be read safely, it stops affected requests. A missing robots file is recorded distinctly from a disallow rule. No alternate identity or browser challenge bypass is attempted.
Fix the intended problem
If a useful public page is unintentionally blocked, review the relevant rule with the site owner. If access is intentionally denied, preserve that policy and obtain the evidence through an approved path. Do not open private pages simply to turn a status green.