AI Bot Verification Tiers

AI Bot Verification Tiers address a fundamental problem in AI crawler analytics: anyone can set any user agent string in an HTTP request. A malicious actor can label traffic 'GPTBot' to confuse analytics, and conversely, some legitimate AI crawlers use non-obvious user agents. Verification tiers provide a confidence level for each attributed crawler request.

The three tiers: Verified means the request has been confirmed as legitimate via reverse DNS lookup (the IP resolves to a hostname in the crawler's official IP range) and ASN matching (the autonomous system number belongs to the crawler's operator). Likely means the user agent matches a known AI crawler pattern but verification wasn't possible or complete. Unknown means the request claims to be an AI crawler but couldn't be verified.

Verification matters because unverified bot traffic inflates AI crawler metrics. A site that appears to receive 10,000 GPTBot requests per month may only have 2,000 verified GPTBot requests - the rest are unidentified scrapers using GPTBot's user agent string. Acting on unverified data leads to incorrect conclusions about AI visibility.

Mentd.'s bot detection system uses reverse DNS lookup and IP ASN matching to classify every server log request attributed to an AI crawler. Only Verified requests are used in core analytics calculations. Likely and Unknown requests are reported separately for transparency.