Home/Campaigns/Web Content Processing

Web Content Processing

When you provide a web page URL as a knowledge base source, ZipTier automatically processes the page through a multi-pass cleanup pipeline. This ensures your AI Assistant learns from meaningful content, not cookie banners, navigation menus, or boilerplate footers. This guide explains what content is retained, what is removed, and how to get the best results.


Why We Clean Web Content

Web pages contain far more than their main content. Navigation bars, cookie dialogs, footer links, and other structural elements can account for over half the text on a typical page. If ingested as-is, this noise degrades your AI Assistant's accuracy and wastes processing resources.

Improve Response Accuracy

By removing irrelevant content, the AI Assistant focuses on the information that actually matters to your prospects.

Minimize Knowledge Base Noise

Navigation menus, cookie policies, and footer links would clutter your knowledge base and confuse responses.

Reduce Processing Costs

Cleaner content means fewer tokens processed per conversation, keeping your AI Conversations efficient.

Prevent Irrelevant Citations

Without cleanup, the AI might cite cookie policies or reCAPTCHA notices when answering prospect questions.


What We Retain

The cleanup pipeline preserves all content that contributes to a rich, informative knowledge base for your AI Assistant.

Content TypeExamples
Headings and body textPage titles, section headings, paragraphs: the main content of the page
Links in the main content"Download the whitepaper", "Our cloud migration guide covers the full process": link text and URL are both kept
Code blocks and technical contentCode snippets, API examples, configuration samples
Tables and structured dataPricing tables, feature comparisons, specification lists
List items with substantive contentFeature lists, step-by-step instructions, bullet-point descriptions

What We Remove

The following elements are stripped during processing because they add noise without contributing meaningful knowledge.

Content TypeWhy It's Removed
Navigation menus and mega-menusRepeated on every page; not part of the page's unique content
Headers and footersCopyright notices, legal links, and social media icons are boilerplate
Cookie consent dialogsLegal compliance UI that has no informational value
Images, thumbnails, icons, bannersVisual elements that cannot be interpreted as text knowledge. Descriptive alt text (longer than 10 characters, not just a filename) is kept as plain text
Breadcrumb navigationSite structure markers, not meaningful content
Form fields and dropdownsCountry selectors, search boxes, and input elements are interactive UI
Boilerplate textPrivacy notices, reCAPTCHA disclaimers, and standard legal copy
Author bio blocksRepeated author cards that appear across multiple blog posts
Table of contents / sidebar sectionsNavigation aids that duplicate heading structure
Duplicate content blocksRepeated CTAs, testimonials, or sections that appear more than once
Garbled UI textRendered mockups or screenshots that produce nonsensical text when parsed
Orphaned UI labelsStandalone button text like "Benefits", "Pricing" with no surrounding context
Generic CTA buttons without URLs"Get started", "Buy now": no destination link, no informational value

Links are handled by where they sit on the page. Links inside the page's main content keep their URLs, so prospects can be pointed to useful resources. Header, footer, navigation and table-of-contents links are removed along with those blocks.

How link decisions are made:Links aren't judged by their wording. A link is kept or removed together with the block it sits in. Here's how different link types are handled:

Link TypeURL Preserved?Example
Links in the main contentYes"Register for the webinar", "See our data security overview for details"
Header, footer and navigation linksNo"About", "Blog", "Contact Us", legal and social links
Table of contents / sidebar linksNo"On this page" jump links that repeat the page's headings

Best Practices

Follow these tips to get the best results when using web pages as knowledge base sources.

Choose the Right Pages

  • Use content-rich pages: Pages with clear headings, structured paragraphs, and substantive text produce the best knowledge base entries.
  • Prefer informational pages over marketing landing pages: Landing pages often consist mostly of images, short taglines, and CTAs with little extractable text.
  • Pages with meaningful links to resources produce richer knowledge: If the page links to whitepapers, case studies, or documentation, those URLs are preserved for reference.

Avoid Common Pitfalls

  • Avoid pages that are mostly images or video: The pipeline processes text content only; image-heavy pages yield very little usable knowledge.
  • Don't rely on content revealed by interaction: JavaScript-rendered pages are generally captured, but content that appears only after a user action (such as a click or a login) may not be.
  • Avoid pages behind authentication: The crawler cannot access gated or login-protected content.

Note:If a web page produces very little content after processing, consider uploading a PDF or document version of the same content instead. Documents give you full control over what the AI Assistant learns.