Loading documentation...
Crawl public websites, preview discovered pages, and ingest content automatically.
Operator features an automated website crawler (src/server/services/crawler/crawler.ts) that reads public business websites and turns pages into searchable knowledge.
Accessed under Knowledge Base > Website Import (/kb), the crawler enables one-click synchronization of your website content.
When initiating a crawl via discoverWebsitePagesAction, you can configure:
- Max Crawl Depth: 1 (target URL only), 2 (target + directly linked pages), or 3 (deep recursive crawl).
- Max Pages: Limit on discovered pages (default: 20, max: 50).
- Include Subdomains: Toggle whether to follow links to subdomains (e.g., blog.yourclinic.com).
- Follow External Links: Defaults to false to stay strictly within your business domain.
- Path Whitelist / Blacklist: Include only specific paths (e.g., /services/*) or exclude sensitive paths (e.g., /cart, /wp-admin).
Crawling is a two-phase process:
1. Discovery Phase: The crawler visits pages and reports back: page title, URL path, word count, estimated chunk count, and suggested category.
2. Selective Ingestion: You review the list, uncheck unnecessary pages (e.g., privacy policy or login screens), and click Ingest Selected Pages. The background pipeline (WebsiteIngestionPipeline) converts each page into a document in knowledge_documents.
To protect server infrastructure, all crawl requests pass through validateSafeUrl in src/server/services/crawler/ssrf.ts:
- Validates protocol (only http: and https: permitted).
- Resolves DNS and blocks loopback, link-local, private RFC 1918 subnets, and AWS/GCP metadata IP 169.254.169.254.