In this guide
Accessible, indexed and cited are three different things
An assistant will not cite your page simply because it responds in your browser. Distinguish access to the URL, whether its content can be read, whether it is indexed, and whether it is selected for an answer. These steps depend on the service and the question asked.
This guide helps you investigate technical barriers on a public page. It does not measure its reputation or calculate an “AI visibility” score. Here, accessibility means crawlers’ access to content; digital accessibility for people is a separate subject to check.
For Google Search’s generative experiences, standard SEO practices remain relevant. A file llms.txt or special markup does not provide a bonus: Google says it does not use them for this purpose. Other services may have their own uses. Official Google recommendations.
1. Check what a signed-out visitor receives
Choose a page intended to be public. Open it in a private window, then inspect the document request in the Network panel. Record the initial URL, redirects, final URL, status and returned content.
- An error
401,403or5xxmust be investigated in context. A block may be intentional. - A status of
200may accompany a sign-in screen, anti-bot challenge or error message. Check the response body. - Compare access from your tools with server or CDN logs. A successful request from your computer does not prove that a crawler receives the same response.
To keep evidence for a URL you administer, replace the example address in this command. It follows redirects and saves headers and body separately:
curl -sS -L -D entetes.txt -o page.html https://example.com/serviceDo not send a sign-in cookie or private token when simulating public access. If the firewall appears involved, identify the rule and the affected crawler before changing the protection. Changing only a test’s User-Agent header does not reproduce a real crawler’s network identity.
2. Compare the initial HTML and the rendered page
In page.html or the source code, look for a phrase from the main content, the title and an important internal link. Compare with the DOM after JavaScript execution in the Elements panel.
Google can execute JavaScript, but crawling, rendering and indexing are separate steps. A blocked script or an error can prevent the expected rendering. Do not assume every assistant has the same capabilities. How JavaScript SEO works according to Google.
For your diagnosis, record what is missing without JavaScript and what only arrives after an interaction. Make essential information and links reliably available. Server rendering or static generation can help, without being a universal requirement for indexing.
A menu visible in a screenshot is not enough: check that important pages have usable links with real destinations. A title, clear sections and factual content also help visitors understand what your page offers.
3. Read the rules without confusing crawling with indexing
Open /robots.txt on the relevant domain. Identify the group matching the crawler and the rules applicable to the path. Its presence does not prove that a page is allowed; its absence alone does not mean crawling is prohibited.
Disallow limits crawling for robots that follow the protocol. It does not protect private content or guarantee that a URL disappears from results. The role and limits of robots.txt.
Inspect separately <meta name="robots"> and the header X-Robots-Tag. Pour que Google lise un noindex, it must be able to access the response containing it. Indexing and snippet directives. Une canonique exprime une préférence de version ; elle ne prouve pas que cette URL est indexée. Canonicalisation signals.
For a public page intended to appear in search, look for unintended restrictions and check discovery links and the sitemap. Keep exclusions for private pages or pages that do not belong in search.
4. Choose search and training permissions separately
“Allow AI” is not a single rule. OpenAI distinguishes the following roles; these do not describe every provider’s behaviour.
On a small screen, scroll the table horizontally.
| Agent | Role | What you should not infer |
|---|---|---|
| Googlebot | Crawling for Google Search. | Successful access does not guarantee indexing or a citation. |
| OAI-SearchBot | Content discovery for ChatGPT search. | This permission is separate from training permission. |
| GPTBot | Collection of content that may be used to train OpenAI models. | Blocking it does not automatically block OAI-SearchBot. |
| ChatGPT-User | Visits triggered by certain user actions. | This is not the automated search crawler; robots.txt may not apply to these actions. |
Search and training controls are independent. For genuinely private access, use authentication and server-side authorisation. Official OpenAI crawler documentation.
Example: allow ChatGPT search and disallow GPTBot
Here is an example for a fictional site, with three excluded areas. It illustrates an editorial choice without recommending it for every site. Do not replace your file without preserving your existing exclusions.
User-agent: *
Disallow: /admin/
Disallow: /compte/
Disallow: /api/
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /compte/
Disallow: /api/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlA specific group does not automatically inherit restrictions from the * : they are therefore repeated for OAI-SearchBot. Read the complete file, including any other groups with the same name. Protocol rules, RFC 9309.
5. Confirm what was observed, without promising a citation
For Google, use Search Console URL inspection if you administer the property. Distinguish Google’s known version from the live test. For other services, compare their documentation with actual access observed in your logs and available monitoring tools.
Argos identifies some public signals: HTTP responses, HTML structure, directives and discovery resources. It does not simulate every crawler, validate your entire robots.txt policy, or know a search engine’s actual index or citation choices.
Document your conclusion at each step: “the response contains the text”, “the rule allows this path for this crawler”, or “the search engine’s tool reports this URL as indexed”. An assistant’s answer to a question can vary; it is not a comprehensive visibility test.
After publication, revisit these points in the redesign acceptance checks, then monitor the results using the available data. Technical access may be a prerequisite; the relevance, evidence and usefulness of the content still need work.
Official references
Sources consulted on 2 November 2026. Service interfaces and rules may change.
Check your page with Argos
Run a free audit of a public URL and examine the available evidence. Automated checks complement the checks in this guide.
Analyse a page for free