Inventory the surfaces
Separate public pages, resources needed for rendering, private areas and variants without independent value.
GUIDE / PROTOCOL
Control AI crawler access
A clear access policy starts with the intended use of each public page rather than a list of crawler names.
Direct answer
How should AI-related crawlers be controlled?
Decide which pages may be discovered, which may appear in search and which should be excluded from a specific use. Express those choices through robots.txt, page directives and server responses, then test what a crawler can actually read.
01 / Review method
Review method
Choose by use
Decide access for conventional search, AI search and training collection using the crawler identities documented by each operator.
Inspect the response
Test HTTP status, robots.txt, redirects, robots meta or X-Robots-Tag and served HTML, including language versions.
Observe carefully
Compare server logs, webmaster tools and referral traffic. A crawler visit proves neither indexing nor citation.
02 / Evidence to keep
Evidence to keep
Dated rule
Keep the version, date and reason for each robots directive.
Tested URL
Record the exact page, status, canonical and page-level rules.
Identified agent
Separate declared search, preview and training crawlers.
Outcome check
Document what remains accessible and what actually appears in the available tools.
PRACTICAL CHECK
Before reaching a conclusion
A frequent misreading and the check that helps avoid it.
Premature conclusion
Blocking a crawler does not automatically remove a known URL from an index or control copies held elsewhere. Rules vary by crawler and purpose.
Decisive check
Read directives actually served on the right host, identify affected crawler categories and test representative URLs. Distinguish crawling, indexing and training use before changing a rule.
AFTER THE REVIEW
Decide from the evidence
One crawler is blocked but the page remains accessible elsewhere
Separate denial to that crawler, indexing by others and reuse of copies. Check the served response and document the actual scope of control.
A changed rule blocks a needed resource
Test affected journeys and crawlers, restore intended access and retain a rollback. Review logs after release.
Decision and review trail
Decision and review trail
What the check must help decide and what to retain so the choice can be reviewed.
Sufficient evidence
Inspect the robots.txt actually served, agent groups, HTTP responses and page directives. Test the relevant search crawler, not a similarly named agent.
Next decision
Record the intended crawl policy and change the narrowest rule. Retest the published URL and monitor configuration changes.
Detailed inspection cases
Typical situations for preparing a check. They do not describe completed assignments or actual observations.
A result remains after blocking
What to inspect
Compare the rule-change date, recent accesses and the result still displayed.
Decision and action
Separate restrictions on new access from removal of already processed content; document each mechanism independently.
Record to retain
Dated rule, available logs and observed result.
The crawler name is insufficient
What to inspect
Inspect the declared agent and validation information published by its provider.
Decision and action
Do not attribute access from User-Agent text alone; retain unverified requests in a separate category.
Record to retain
Time, path, declared agent and validation outcome.
A rule covers too many paths
What to inspect
Test intended exact paths and adjacent paths after changing robots rules.
Decision and action
Match the declared intention; do not treat robots as access control.
Record to retain
Declared agent, tested paths, rule and served response.
Review before deciding
These questions complement the method. They help establish what is actually supported while retaining qualifications in the record.
Rule and scope
Does the published rule target the right agent and paths? Distinguish a crawling declaration, access control and actual observations in available logs.
View a worked decision
Fictional example: the state applies only to the criterion and scope described.
- Fictional observation
- A Disallow rule targets Robot-A on /private/; no log allows this robot to be observed.
- Declared state
- Not checked
- Decision and next check
- Document the published rule; leave actual behaviour unknown. Do not present it as an access barrier.
Opposite controls
Have an allowed and an excluded path been compared without changing access? Revisit rules when an operator changes its agents or documentation.
View a worked decision
Fictional example: the state applies only to the criterion and scope described.
- Fictional observation
- In the test set, /public/ is allowed and /private/ excluded for the same agent; both matches are checked.
- Declared state
- Verified within this scope
- Decision and next check
- The check covers these rules and paths. Repeat controls if the agent name or rule changes.
Sensitive resource
Does a private resource require authentication independently of robots.txt? A crawling instruction is not access protection.
Documentation consulted : IETF · Robots Exclusion Protocol ·
View a worked decision
Fictional example: the state applies only to the criterion and scope described.
- Fictional observation
- The test resource /private/ appears in robots.txt but remains readable without authentication.
- Declared state
- Observed discrepancy
- Decision and next check
- Record the missing expected protection and fix access control. Do not publish sensitive content for testing.
TOOLS / REGISTERS
Information to record
Retain the exact crawler name and tested path. The published rule and observed HTTP response are separate evidence; check both after every change.
- Access date and time zone
- Exact URL
- Interface / mode / account
- Observed robots directives
- HTTP status / redirect
- Decision / next action
- Review owner
- Next review
03 / FAQ
Frequently asked questions
Does allowing OAI-SearchBot guarantee a citation?
No. It enables access needed for possible appearance in ChatGPT search but does not guarantee use of a page.
Can search and training be treated separately?
Yes when an operator documents separate agents. Verify names and behaviour in official documentation before changing the rules.
Does robots.txt protect a private page?
No. Use appropriate access controls; robots.txt addresses crawlers that choose to honour the rule.
Should images be blocked separately?
Check intended use and resources needed for rendering before adding a rule; blocking can reduce page comprehension.
How is a change verified?
Test the served file, relevant user-agent group, representative URLs and logs after release.
04 / Official references
Official references
05 / Continue reading
Continue reading
Describe a context
DOCUMENTED REFERENCE
What robots.txt actually describes
Separate a published rule from the response actually received by a crawler. For an important path, retain the consulted file and the URL check.
| Reference | Value or form | What it identifies |
|---|---|---|
| User-agent | group | Target crawler group. |
| Allow | path | Allowed path. |
| Disallow | path | Path excluded from crawling. |
Implication for the record
The protocol is not an access-authorisation mechanism. Do not protect a secret with Disallow alone. These fields describe the protocol; separately check each platform’s documented agents and policies.
Primary documentation
Documentation consulted on .
DEEPER READING / PRACTICE
Read a rule of access in its context
A team wants to understand which robots can consult certain resources. The examination should focus on the published rules and their scope, and then on the observations of available requests.
Updated on .
Priority checks
Read the guidelines groups, the routes concerned and the differences between stated agents. Check that the rules match the URLs that are actually served and keep their version.
Interpretation pitfalls
A published directive does not demonstrate that a visitor respects it. The name of an agent in a request is also not enough to establish the identity of its operator.
Working record to retain
The report describes the rule, path, declared agent and expected effect. Log observations remain separate from the intention expressed in the guidelines file.
PRACTICAL CASE / WORKED EXAMPLE
A rule does not cover the URLs actually served
The example is a web control educational situation. It connects the defect to a concrete path and describes a way to verify the correction, rather than consider the only publication of the change as a proof of resolution.
Example context
A guideline file targets a old path, while content is accessible under a new structure. The intention published no longer corresponds to the current scope.
Suggested approach
Compare effective rules and paths. Examine agents groups, URL variants and available observations. Keep the version before correction to explain the change.
Conclusion and next step
The report describes the gap between rule and resources. The control distinguishes the extent of the directives and the behavior of the requests observed after their modification.
Questions for deeper practice
Can a compliance with the rules be deducted from the file alone?
Not to. The file describes an intention of access. Request observations and their assignment require a separate reading, with the identification limits declared by visitors.
DETAILED WORKSHOP / APPLICATION
Compare access directives with current URL paths
01 / Prepare the necessary information
Preparing a file of guidelines, agents groups and URL structure. Identify the targeted resources and variants that may change the described access scope.
02 / Carry out the approach
Compare each rule to the roads effectively served. Release the exceptions and keep the file version. Examine separately the observed requests and the limits of their assignment.
03 / Examine an alternative situation
If a rule describes the intention well but requests persist, do not confuse publication and respect. Observed behavior requires a separate analysis of the identifiers and resources requested.
04 / Decide on the next step
The inspection of the directives is closed when the published scope corresponds to the documented intention. The verification of visitors and their behavior remains qualified separately.
05 / Hand over the verification records
Transmit rules, paths, observations and reservations. A change in trees should trigger a review of the directives that depend on the former locations.
ANNOTATED WORKING DOCUMENT / REVIEW GRID
A report of the directives and their effective scope
The report combines declared agent, rule, path and intention. It keeps the published version and observations of requests in a separate reading of compliance with the directives.
On a small screen, scroll the grid horizontally to read all three columns.
| Observation | Supported interpretation | Next step |
|---|---|---|
| The rule concerns the roads currently served. | The published scope corresponds to the documented intention. | Keep the version and check the important URL variants. |
| A rule applies only to an old location. | The publication does not necessarily cover the current resources. | Check the routes and then check the groups and exceptions. |
| There is a number of requests in spite of a directive. | The file alone does not allow conclusions about the identity or behavior of the visitor. | Examine the observations and their assignment limits separately. |
Reviewing the working document
The person responsible for verification shall verify that the statement describes an intention of access or a fact observed. The two should not be confused, even when the name of an agent seems to match a rule.
REPORTING / EXAMPLE WORDING
Presentation of results: Read a rule of access in its context
Point to clarify
A directive expresses a planned scope without proof of visitor behavior.
Before handing over
Check that the formulation distinguishes published intention and request observation.
Example of a qualified conclusion
The published file describes the rules and paths indicated. Observed requests are analyzed with their assignment limits and are not deducted from the file alone.