Review robots, sitemaps, canonicals, and discovery rules
Find the crawl and indexing controls that make important pages harder to discover or trust.
Fix
Crawl Controls helps you review robots.txt, sitemaps, canonical rules, and redirect paths that can affect discovery and technical clarity. Use it to remove the blockers that make pages harder to crawl or interpret.
What this tool does
- Review discovery and crawl-control signals
- Find blockers in robots, sitemaps, canonicals, and redirects
- Use the results to improve technical discoverability
Fix
Where This Fits
Crawl Controls is a Fix tool. Use it when crawl directives, indexing rules, or sitemap coverage may be weakening visibility.
What you see in the product
- A structured view of crawl-control issues and affected areas
- Severity guidance for technical discovery blockers
- Fix direction for robots, sitemaps, canonicals, and redirect problems
Plan behaviour
Plan behaviour
- Free — Useful visibility into crawl-control issues where surfaced
- Pro — Stronger operational depth and reporting around crawl-control work
- Agency — Broader delivery and scaled workflow support
When to Use It
When to Use It
- You suspect engines are missing pages they should be able to discover.
- You are reviewing crawl directives after a technical change.
- You need to check discovery hygiene before deeper content work.
Common Pitfalls
- Treating crawl-control issues as minor when they block discovery.
- Reviewing only sitemaps while missing canonical or redirect problems.
- Making crawl-rule changes without checking the affected pages afterwards.
Example
Example
You use "Crawl Controls" to identify the blockers that matter most, work through the recommended actions, and then return later to confirm whether the issue was resolved.
What it looks like
Crawl Controls tune scope, cadence, and agent behavior.
- Scope settings
- Scheduling
- Agent options
Next steps
Start with the plan that fits your current stage, or compare tiers if you need more reporting depth, outputs, or workflow scale.