Skip to main content
Edition: Community Crawl brings public or allowed website content into your project so AI Search and AI Chatbot can cite it. In the console Sources screen, website content comes from two tabs: Domain and Sitemap XML.

Domain

Add a website or documentation URL and choose how deep to follow links — from This page only up to 5 levels (2 levels is recommended).

Sitemap XML

Train on every page listed in a sitemap.xml. Enter the sitemap URL (for example https://example.com/sitemap.xml).
  • Sitemap index files and .xml.gz files are supported.
  • Only pages listed in the sitemap on the same site are trained.
Use a sitemap when your site already publishes one and you want exactly those pages — without depending on link depth.

Guidance

  • Prefer authoritative sources your organization owns or is licensed to use.
  • Retrain when source sites change materially, or set a daily or weekly schedule.
  • Verify citations after large crawl jobs before publishing widgets widely.

Interactive tour