Web Content Processing
When you provide a web page URL as a knowledge base source, ZipTier automatically processes the page through a multi-pass cleanup pipeline. This ensures your AI Assistant learns from meaningful content, not cookie banners, navigation menus, or boilerplate footers. This guide explains what content is retained, what is removed, and how to get the best results.
Why We Clean Web Content
Web pages contain far more than their main content. Navigation bars, cookie dialogs, footer links, and other structural elements can account for over half the text on a typical page. If ingested as-is, this noise degrades your AI Assistant's accuracy and wastes processing resources.
Improve Response Accuracy
By removing irrelevant content, the AI Assistant focuses on the information that actually matters to your prospects.
Minimize Knowledge Base Noise
Navigation menus, cookie policies, and footer links would clutter your knowledge base and confuse responses.
Reduce Processing Costs
Cleaner content means fewer tokens processed per conversation, keeping your AI Conversations efficient.
Prevent Irrelevant Citations
Without cleanup, the AI might cite cookie policies or reCAPTCHA notices when answering prospect questions.
What We Retain
The cleanup pipeline preserves all content that contributes to a rich, informative knowledge base for your AI Assistant.
| Content Type | Examples |
|---|---|
| Headings and body text | Page titles, section headings, paragraphs: the main content of the page |
| Links in the main content | "Download the whitepaper", "Our cloud migration guide covers the full process": link text and URL are both kept |
| Code blocks and technical content | Code snippets, API examples, configuration samples |
| Tables and structured data | Pricing tables, feature comparisons, specification lists |
| List items with substantive content | Feature lists, step-by-step instructions, bullet-point descriptions |
What We Remove
The following elements are stripped during processing because they add noise without contributing meaningful knowledge.
| Content Type | Why It's Removed |
|---|---|
| Navigation menus and mega-menus | Repeated on every page; not part of the page's unique content |
| Headers and footers | Copyright notices, legal links, and social media icons are boilerplate |
| Cookie consent dialogs | Legal compliance UI that has no informational value |
| Images, thumbnails, icons, banners | Visual elements that cannot be interpreted as text knowledge. Descriptive alt text (longer than 10 characters, not just a filename) is kept as plain text |
| Breadcrumb navigation | Site structure markers, not meaningful content |
| Form fields and dropdowns | Country selectors, search boxes, and input elements are interactive UI |
| Boilerplate text | Privacy notices, reCAPTCHA disclaimers, and standard legal copy |
| Author bio blocks | Repeated author cards that appear across multiple blog posts |
| Table of contents / sidebar sections | Navigation aids that duplicate heading structure |
| Duplicate content blocks | Repeated CTAs, testimonials, or sections that appear more than once |
| Garbled UI text | Rendered mockups or screenshots that produce nonsensical text when parsed |
| Orphaned UI labels | Standalone button text like "Benefits", "Pricing" with no surrounding context |
| Generic CTA buttons without URLs | "Get started", "Buy now": no destination link, no informational value |
Link Handling
Links are handled by where they sit on the page. Links inside the page's main content keep their URLs, so prospects can be pointed to useful resources. Header, footer, navigation and table-of-contents links are removed along with those blocks.
How link decisions are made:Links aren't judged by their wording. A link is kept or removed together with the block it sits in. Here's how different link types are handled:
| Link Type | URL Preserved? | Example |
|---|---|---|
| Links in the main content | Yes | "Register for the webinar", "See our data security overview for details" |
| Header, footer and navigation links | No | "About", "Blog", "Contact Us", legal and social links |
| Table of contents / sidebar links | No | "On this page" jump links that repeat the page's headings |
Best Practices
Follow these tips to get the best results when using web pages as knowledge base sources.
Choose the Right Pages
- Use content-rich pages: Pages with clear headings, structured paragraphs, and substantive text produce the best knowledge base entries.
- Prefer informational pages over marketing landing pages: Landing pages often consist mostly of images, short taglines, and CTAs with little extractable text.
- Pages with meaningful links to resources produce richer knowledge: If the page links to whitepapers, case studies, or documentation, those URLs are preserved for reference.
Avoid Common Pitfalls
- Avoid pages that are mostly images or video: The pipeline processes text content only; image-heavy pages yield very little usable knowledge.
- Don't rely on content revealed by interaction: JavaScript-rendered pages are generally captured, but content that appears only after a user action (such as a click or a login) may not be.
- Avoid pages behind authentication: The crawler cannot access gated or login-protected content.
Note:If a web page produces very little content after processing, consider uploading a PDF or document version of the same content instead. Documents give you full control over what the AI Assistant learns.