Websites & Crawling

Adding sites, validation, how crawls work, sitemaps and robots.txt.

1. 1Adding and managing websitesManage all the websites in your workspace from the Websites page, including settings per site.
2. 2How crawling worksWhat the crawler reads on each page, how it discovers URLs, and which rules it obeys.
3. 3Starting a crawl and monitoring progressStart, watch, and stop crawls from the Technical SEO page with live status updates.
4. 4Crawl history and errorsReview past crawl runs, understand their statuses, and diagnose failed crawls.
5. 5Sitemap discovery and robots.txt handlingHow AI SEO Manager finds your sitemap, how robots.txt is honored, and the settings that control both.
6. 6Understanding crawled pagesRead the pages table and the per-page detail: status, indexability, words, inlinks, and metadata.