How to Govern AI Traffic on Your Website

Consider a hypothetical online store you manage. A shopping assistant, working for a customer, reads a product page for running shoes, checks the size chart, and compares two colors. It makes a few requests and leaves. Minutes later, another automated visitor repeatedly hits your login page with different passwords, then probes a configuration path no shopper would open.

Both visitors are automated. Both may claim the same company name. Your rules should evaluate what each visitor is trying to do, the evidence for its identity, and the permission the action requires.

Know What Is Visiting Your Site

Automated visitors fall into rough groups. A crawler collects pages for an index or a model. An assistant fetches information for a person who asked a question. An agent tries to act, such as adding an item to a cart or submitting a form. These roles can overlap, and one company can operate all three. Your server sees requests, so a visitor’s purpose needs to be assessed from the available evidence.

Start with the request. Before asking who is calling, look at what is being requested. Reading a public product page carries less risk than viewing order history or submitting a payment. Sorting traffic by requested action gives you a stable basis for access rules: a visitor changing its name shouldn’t change the permission an action requires.

Treat a name as a claim. The user-agent string is a label the visitor writes for itself. GreyNoise research reported in August 2026 found scanners using the crawler names of OpenAI, Anthropic, Google, Perplexity, and others across 824 IP addresses while searching for exposed credential files. That observation illustrates why a familiar label is insufficient evidence of identity.

Cloudflare’s Web Bot Auth documentation describes cryptographic signatures in HTTP messages as a way to verify automated requests. A successfully verified signature provides stronger evidence about the sender than a self-declared label. Permission to perform an action still needs a separate check. Assessments based on network signals or behavior also carry uncertainty, especially when used to infer intent.

Match Access to the Action

Authentication checks whether a visitor is who it claims to be. Authorization determines whether that visitor may perform the requested action. OWASP’s Authorization Cheat Sheet distinguishes these checks, and its Authorization Patterns guidance recommends denying by default and validating permissions on every request.

A signed request from a known assistant operator can pass an identity check and still be refused access to a customer’s account. The operator’s identity does not establish that customer’s permission.

This table offers an illustrative policy sketch for a store. Adapt the checks and responses to your application rather than treating them as universal rules or vendor defaults.

Allow useful reading. An assistant fetching a public size chart for a shopper is using the page as intended. You can allow that activity while watching its volume and effect on the site.

Limit risky volume. Rate limiting sets an allowance of requests per visitor over a period, then slows or pauses traffic above it. It can reduce server load and password-guessing attempts without requiring certainty about intent.

Check permission for sensitive actions. Viewing an order, changing an address, or paying should depend on a valid customer session and appropriate permission. Validate access on every request, and require additional confirmation when the action’s risk warrants it.

Responses can include allowing, monitoring, constraining, or blocking traffic. OWASP’s bot-management guidance recommends graduated responses rather than relying on hard blocks alone. The response should fit the evidence and potential harm; clear abuse need not pass through every less restrictive option first.

For a commercial example, CHEQ describes agent trust management as a way to classify visitors using behavioral and network signals and an independent identity graph. It also describes inferring likely intent from behavior and applying customer-defined policies, including allowing, monitoring, stepping up verification, throttling, or blocking. The company presents this assessment as a complement to credential-based authentication, with inferred intent treated as an assessment rather than certainty.

Roll Out Controls Without Breaking Real Visits

Map sensitive routes. List the paths where unauthorized activity could cause harm: login, account pages, checkout, admin screens, and connected APIs that write data. Pair traffic detection on these routes with permission checks inside the application. OWASP warns that a single bot-management control is brittle, and its REST Security guidance cautions against relying exclusively on API keys for sensitive or high-value resources.

Keep crawler preferences separate from access controls. Google documents that robots.txt rules may not be supported by all crawlers. The file can communicate preferences, but private pages still need authentication and authorization.

Observe before tightening. Run proposed traffic rules in monitor mode, where available, to see which requests they would affect. Build a baseline of normal activity and test against known legitimate sessions, including any assistant integrations you rely on. Keep existing authentication and permission checks active during this testing.

Check the application too. A service filtering traffic before it reaches your application can classify a visitor, but the application must check whether the session may access a particular order. Keep those permission checks close to the protected data. When evaluating any traffic-management service, test its policies against legitimate assistant traffic and confirm how it deploys with your specific stack.

Review mistakes. Record enough context to establish when an event happened, where it happened, who was involved, and what occurred, as OWASP’s Logging Cheat Sheet recommends. Useful fields include the time, route, available identity evidence, requested action, decision, and reason. Keep passwords, access tokens, and unnecessary personal data out of the log.

Review unexpected denials on a regular schedule and keep a rollback path for rules that block legitimate visits. Pilot new traffic policies on one sensitive route, such as login, and review the results before expanding enforcement. Connected APIs deserve the same care, and established API security practices such as per-client rate limits and behavioral monitoring apply directly to agent traffic.

Conclusion

Good AI traffic governance starts with the action, not the label. Verify identity where possible, check permission for every sensitive request, and use proportionate limits for uncertain or risky behavior. Review logs and adjust rules when legitimate visits are affected.

FAQs

Should I block every automated visitor?

No. Blanket blocking can remove useful assistants and crawlers along with abusive traffic. Allow appropriate public access, monitor volume, and constrain or block activity according to its behavior and the risk of the requested action.

Can I trust a browser’s user-agent label?

Treat it as a claim rather than identity proof. Use other evidence, such as verified signatures and behavioral signals, to assess the visitor. Then check whether it has permission for the requested action.

Does robots.txt protect private pages?

No. Crawlers may ignore its rules. Use it to communicate crawling preferences, and protect private data with authentication and authorization.

When should a request need extra approval?

Sensitive requests always need valid permission, but they don’t all need a new approval prompt. Consider additional verification for transactions or account changes when the risk warrants it. Continue checking access permissions on every request.