Most AI visibility checklists start with schema because audit tools put schema on the report. I start with robots.txt because bots request access before they can judge a page. On our own site, robots.txt drew 3,867 verified AI hits and 1,222 live-answer fetches, more than the homepage in the same 30-day window I logged in our crawler teardown. A mistake there costs more than a schema gap: no fetch means no answer to lift, entity to resolve or update date to notice. Run these 21 checks in order. Stop at the first failure.

Tier 1: Let the bot through the file, firewall and server

AI bot stopped at a robots.txt gate before reaching website pages

  1. Allow OAI-SearchBot. Pass/fail: its robots.txt rules do not block the pages you want in ChatGPT search answers. OpenAI assigns its crawlers different jobs: OAI-SearchBot serves search, while GPTBot is for training. You can allow search without allowing training. Check the actual rule group rather than assuming one decision covers both; if you change it, allow roughly 24 hours for the robots.txt change to propagate.
  2. Allow PerplexityBot; handle Perplexity-User separately. Pass/fail: the indexing bot is permitted and can get through your server or WAF. PerplexityBot builds the index used for surfacing pages; it is not the training bot. Perplexity-User generally ignores robots.txt because a person requested the page. If you need to control those user-triggered fetches, do it at the server or WAF, not with a rule they may not follow. Check Perplexity’s published IP guidance when configuring access.
  3. Allow ClaudeBot; keep Google-Extended in its own lane. Pass/fail: no broad Disallow: / hides under an AI-labeled rule you meant to use more narrowly. Google-Extended is a robots.txt token for Gemini training choice, not a separate crawler; Googlebot still handles ordinary crawling. Set Google-Extended according to your training preference without confusing it with Search or AI Overviews access. Keep Googlebot allowed, and use the AI crawler swipe file rather than improvising rule blocks.
  4. Check the edge before you blame robots.txt. Pass/fail: permitted bot traffic gets a 200, not a 403 or a challenge, when it requests the homepage. A user-agent request is a useful first check, but read your edge and server logs too; typing a bot name into a header does not make your request that bot. Cloudflare announced AI-crawler blocking changes on July 1, 2025, and Perplexity publishes WAF whitelisting guidance. Check Cloudflare’s Security > Bots and AI Crawl Control, or AWS WAF’s managed bot rules and IP sets. A clean file cannot overrule a blocking firewall.
  5. Get money pages to 200 without a detour. Pass/fail: pricing, product and contact pages return 200 in fewer than two redirects, without an interstitial or login wall blocking the fetch. ChatGPT-User is user-triggered, so robots.txt rules may not apply. That makes the response from your server the check that matters. Strip redirect chains, remove the country-splash redirect for bot traffic and serve the page you want read.

Tier 2: Make the answer page discoverable

  1. Keep sitemap.xml small, fresh and honest. Pass/fail: it returns 200, contains only indexable 200 URLs and reports a lastmod date that matches a real, visible update. Drop redirects, noindex pages and PDFs you do not want cited. Regenerate the file when you publish. A sitemap full of destinations that do not resolve cleanly sends the bot back into the access problems you just fixed.
  2. Link answer pages from pages bots already fetch. Pass/fail: every page answering a buyer question has a crawlable <a href> from the homepage, pricing page, main category or top blog index. Use a contextual anchor that says what question the destination answers, not a pile of links in the footer. Access does not guarantee discovery: the bot still needs a route from a page it knows to the page you want cited.
  3. Find the orphans before editing their copy. Pass/fail: check site:yourdomain.com and your server logs for each target URL, including whether it was fetched in the last 30 days. A missing log hit in that window is a reason to investigate, not proof that the page can never appear. Add an internal link, put the URL in the sitemap and request indexing in Search Console once. Do not work through items 9 to 21 on an orphan while its route into the site remains broken.

Tier 3: Put the answer in HTML the bot can read

  1. Server-render the main answer. Pass/fail: disable JavaScript, reload and read the answer in the page’s raw HTML. Server-render the article body, pricing facts and comparison tables; a client-side React shell that delivers the text only after JavaScript runs is a bad bet. The crawler analysis discussed here found a JavaScript-rendering gap among major AI crawlers. Test View Source, not just Inspect Element: the latter can show content your browser added after the initial response.
  2. Put key facts in selectable text, not only images. Pass/fail: every number or comparison you want quoted appears in HTML text. Keep the chart or screenshot for people, then write its important facts underneath it. Alt text is useful, but it is not a replacement for the answer itself. Content that needs JavaScript to reveal, including some tabs and accordions, poses the same reading problem: if the fact is missing from the initial page, do not expect the bot to retrieve it.
  3. Render tab and accordion text on first load. Pass/fail: View Source contains the full text of every panel you need a bot to read. Put the panels in the initial HTML, use JavaScript only to collapse them visually and give sections anchor links where useful. A polished interactive tab is not helping if the answer exists only after a click. Fix the empty source before you rewrite the copy inside it.

Tier 4: Give it a passage worth citing

  1. Lead target sections with a direct answer. Pass/fail: the first sentence after the question heading answers that question without setup. Aim for a self-contained passage of roughly 40 to 60 words, then put the background underneath. If readers need the previous paragraph to decode the answer, the passage is harder to lift on its own. Audit tools may send you to schema first; give the bot a usable sentence first.
  2. Give each buyer intent one clear URL. Pass/fail: a site:yourdomain.com search for the question shows one obvious answer page, not three near-duplicates competing to say the same thing. Merge overlapping posts, redirect the ones you retire with a 301 and keep one live URL for that question. Duplicates split the attention of a bot trying to choose what to fetch and cite. This is consolidation, not permission to bury different buyer questions on a single page.
  3. Use real lists and tables for enumerated facts. Pass/fail: steps, prices, pros and cons, and comparisons use native <ol>, <ul> or <table> HTML rather than styled divs or screenshots. Rewrite a comparison as a three-column table with the verdict in its cells when that is the clearest shape. Structured HTML keeps the relationships between facts visible and makes a passage easier to carry into an answer. Do not make the bot reconstruct a table from a picture.
  4. Put the claim on the landing page you want cited. Pass/fail: its definition, price or step appears in text there, not only in a linked PDF or video. Monitoring software may show that the landing page was skipped; it will not move the missing sentence onto it. Make the indexable page answer the question directly. If a person must click twice to find the fact, you have given the bot a longer route than it needs.

Tier 5: Make the source unmistakable

Small business storefront with mismatched name signs confusing a robot reader

  1. Use one consistent Organization identity. Pass/fail: the homepage schema, footer, contact page and Google Business Profile use the same business name. Keep address and phone formats consistent, and keep sameAs links aligned with the profiles you claim. I used to tell clients schema was decoration. I was wrong about its usefulness here: it gives the bot another way to connect the answer to the business behind it. Do not make that connection harder with conflicting names.
  2. Give each author a real page. Pass/fail: each byline links to an /author/name page with a photo, role, three prior posts and an external credential. Replace “Admin” bylines and bios written like keyword paragraphs. On local and B2B pages, connect the author to the business rather than a freelancer marketplace. The point is to make attribution legible, not to build an elaborate author page while the article body still arrives as an empty shell.
  3. Link to the entities you claim. Pass/fail: your about page links to the review profiles, certifications and suppliers it mentions, using matching names. If you say you install Trane systems in Austin, link Trane and your license page. Give the bot a clear route between the claim and the entity behind it. One coherent identity is more useful than five half-filled profiles that disagree about who you are.

Tier 6: Update the page the bot already knows

  1. Match the visible update date to lastmod. Pass/fail: the date a person sees agrees with the date in the sitemap. Put dateModified and a rendered date near the H1, then change them when the page actually changes. Access and rendering decide whether the page can be read at all; freshness matters only after that. In the Ahrefs 17M-citation analysis, AI Overviews cited pages about 16 days older on average than organic results, while ChatGPT skewed fresher. Do not treat a fresh timestamp as a substitute for a reachable answer.
  2. Add a short changelog to buyer pages. Pass/fail: pricing, comparison and how-to pages say what changed and when. Two plain lines will do; if a plan or limit changed, name the change rather than merely stamping today’s date on the page. Readers checking a consequential fact should not have to guess whether the copy or just the timestamp is new. Keep the note close enough to the answer that it explains the update.
  3. Refresh the winner; do not publish a rival. Pass/fail: one live URL answers the question, and you update that URL in place. Keep the useful heading, revise the answer and note what changed. A near-duplicate starts its own discovery path while the page bots already fetch sits stale. If item 13 found competing posts, consolidate them first; this check is what keeps the same problem from returning on your next update.

Before you buy another visibility dashboard

Leave polish until access, discovery, rendering and extraction pass. That means holding off on extra Organization markup, FAQ schema tuning, readability scores and authority scores while Tiers 1 to 4 still fail. None of those extras opens a blocked route or fills an empty HTML shell. A dashboard that monitors AI Overviews can show which prompt you lost. It cannot, by itself, repair the reason the bot never fetched the answer page.

I grade a tool on who does the fix. groas runs technical fixes alongside new pages inside its SEO and AI Search Sprint, with actions logged, rather than handing you another list of problems to manage. For this checklist, buy execution first and monitoring second.

The item most often skipped is 4: check the edge before you blame the file. Verify what bot traffic receives, not just what your own browser sees. Skip that check and the cost is ugly: content, schema and dates can all be ready while Cloudflare or AWS WAF keeps the bot out for months. I check Tier 1 before I touch a heading. You should too.