IAB Tech Lab, the standards‑setting body behind the OpenRTB protocol and ads.txt, unveiled a new “Bot and Crawler Management Guidance” on May 27, 2026. The document, now open for public comment until June 25, offers a menu of practical strategies for publishers, SSPs, and DSPs to identify, classify, and control non‑human traffic as the industry rolls out the first version of the CoMP (Crawl‑Optimization & Management Protocol) API. By translating a technically dense problem into a set of decision‑making levers, the guidance aims to lower operational friction, curb AI‑driven content scraping, and protect the economics of the emerging content‑marketplace.
What the Guidance Covers
The guidance breaks bot‑management into three core levers: visibility, classification, and action. Visibility recommends telemetry stacks that surface real‑time bot signatures; classification provides a taxonomy ranging from “friendly crawlers” (e.g., Googlebot) to “extractor bots” that harvest copyrighted assets for AI training. Action outlines graduated controls—rate‑limiting, challenge‑response, and outright denial—aligned with the CoMP API’s botPolicy field. By framing the problem in business‑oriented questions (“What value does this traffic deliver?”), the document helps non‑technical leadership assess cost‑benefit trade‑offs without deep engineering expertise.
Why It Matters for Publishers and AI
Bot traffic now eclipses human visits on many news sites. Gartner predicts that by 2025 bots will represent 40 % of all web traffic, siphoning bandwidth and inflating latency. Simultaneously, large‑language‑model agents scrape open‑web content to train generative AI, raising intellectual‑property concerns that traditional ad‑fraud filters miss. The new guidance gives publishers a playbook to differentiate “value‑adding allies” (search indexers, legitimate data partners) from “resource‑draining extractors,” thereby preserving ad inventory quality and safeguarding revenue streams.
Industry Context and Competing Approaches
Commercial bot‑management solutions—Cloudflare Bot Management, Akamai Bot Manager, and Google reCAPTCHA Enterprise—focus on network‑level detection and challenge mechanisms. While effective for high‑volume attacks, they rarely expose the policy granularity needed for CoMP’s intent‑based API calls. IAB Tech Lab’s approach is standards‑first: it embeds bot classification into the same metadata that powers programmatic buying, enabling SSPs and DSPs to bid on inventory with explicit bot‑risk signals. This contrasts with siloed vendor tools that require separate contracts and data pipelines.
Implications for Enterprise Marketing Teams
For brand‑safety and performance teams, the guidance translates into clearer KPI attribution. When a campaign’s viewability drops because an AI scraper inflates impression counts, the CoMP API can now surface a botScore that marketers can filter out in real time. According to Forrester, organizations that implement structured bot‑filtering see up to a 30 % reduction in ad‑fraud losses. Enterprise marketers using CDPs or DMPs can ingest these signals to refine audience segments, ensuring that first‑party data remains uncontaminated by synthetic traffic.
Next Steps and Public Comment
IAB Tech Lab invites stakeholders to review the draft at https://iabtechlab.com/botmanagement/ and submit feedback by June 25. The lab plans to incorporate community input before publishing the final version alongside the CoMP API v1. Adoption will likely be driven by major ad‑tech platforms that already expose bot‑risk fields in their bid‑request schemas, such as The Trade Desk and MediaMath. Early adopters could gain a competitive edge by offering advertisers transparent, bot‑clean inventory—a differentiator increasingly prized by privacy‑conscious brands.
Market Landscape
The bot‑management market sits at the intersection of cybersecurity, AI governance, and programmatic advertising. IDC estimates the global bot‑mitigation market will exceed $7 billion by 2027, propelled by rising AI‑generated content and stricter privacy regulations like the EU’s Digital Services Act. While large cloud providers bundle basic bot detection into their CDN services, the need for a unified, industry‑wide taxonomy remains unmet. IAB’s guidance, anchored in the CoMP API, could become the de‑facto reference model, much as ads.txt standardized seller‑domain verification a decade ago.
Top Insights
- Standardized taxonomy: The guidance introduces a common language for “friendly,” “neutral,” and “extractor” bots, enabling cross‑platform policy enforcement.
- CoMP integration: By mapping bot classifications to the CoMP API’s
botPolicyfield, publishers can automate access decisions without bespoke engineering. - Revenue protection: Gartner’s 40 % bot‑traffic projection underscores the financial upside of precise bot filtering for ad‑tech ecosystems.
- Enterprise relevance: Forrester’s 30 % fraud‑loss reduction benchmark shows measurable ROI for brands that adopt structured bot signals.
- Community‑driven evolution: Open public comment ensures the guidance reflects real‑world publisher constraints and avoids vendor lock‑in.
Get in touch with our Adtech experts
