What is crawl?

A crawl is an import that starts at one web address and follows the site's links to collect pages for the knowledge base. It is the fastest way to give an AI support agent your public content, because the pages are already written for customers and already public. A sitemap import is the near relative: it reads the pages listed in sitemap.xml instead of following links, so it reaches pages no menu links to.

Where crawls differ: the page limit, the depth (how many links away from the start the crawler goes), whether robots.txt is honored, whether only the main content is kept or the navigation and footer come along, how often pages are re-fetched, and whether a re-fetch re-indexes every page or only the ones that changed. The last two decide whether a price change on your site reaches the agent on its own or waits for someone to click.

Where it runs out: a crawler sees what an anonymous visitor's first request sees. Pages behind a login or a cookie wall, pages rendered only by JavaScript, and content inside images or PDFs linked from a page are invisible or incomplete, and those go in as uploads. A crawl of a site with stale pages imports the stale pages too, so the agent inherits every contradiction the site contains, and the first job after a crawl is reading what came in.