llms.txt and AI Crawler SEO: What Actually Works

Does llms.txt actually get a site cited by AI?

No — not on any evidence that currently exists. llms.txt is a proposal, not a standard: no W3C status, no IETF status, no RFC, and as of mid-2026 no major answer engine has confirmed that it reads the file. Every article calling it the new robots.txt for AI is describing an intention, not a measured behaviour.

The measurements point the other way, and they are large. Ahrefs analysed more than 137,000 domains in May 2026 and found that 97% of llms.txt files had never been fetched by any AI crawler. SE Ranking's November 2025 study across 300,000 domains found no measurable improvement in AI citations for sites that published one. Cyrus Shepard's May 2026 meta-analysis of 54 experiments and patents scored llms.txt lowest of the 23 AI-visibility factors it ranked. Google's own AI optimisation guide (Search Central, 2026) is blunter still.

So why serve one at /llms.txt? Because "neither harm nor help" is a cheap trade. It takes ten minutes to generate from data the router already has, and it has real consumers: IDE agents, MCP clients, and audit tools checking whether you have one. It is also a portfolio artifact — anyone assessing my AI crawler and GEO readiness work can fetch it and judge for themselves. That is a good reason to ship one, not a reason to believe it drives citations.

How do AI answer engines actually find your pages?

They retrieve from search indexes they mostly did not build. Google's AI Overviews and AI Mode retrieve from the Search index — no separate AI crawler feeds them. ChatGPT's live search leans heavily on Bing. Perplexity and Claude run their own retrieval crawlers. For none of them is there a submission form, a console or a verification step.

So the nearest thing to a push channel for AI answers is not an AI product. IndexNow pushes changed URLs into Bing, Yandex, Seznam and Naver, and Bing is the index under Copilot, DuckDuckGo and ChatGPT live retrieval. Google has no equivalent open endpoint. Any tool offering a "submit to ChatGPT" button is reselling a crawl.

So ai crawler seo is not a separate discipline bolted onto search. Everything deciding whether you are in those indexes — whether the crawler gets a 200, whether the text is in the HTML, whether the canonical resolves — now gates two channels instead of one. Split it into an "SEO" invoice and an "AI" invoice and you buy the same fix twice.

What is llms.txt, and what is it not?

llms.txt is a plain-Markdown file at the root of a domain giving a language model a curated map of the site: a title, a one-line summary, and sections of annotated links. Jeremy Howard published the proposal in September 2024; it is documented at llmstxt.org. A companion convention, /llms-full.txt, inlines the full text of those pages for single-fetch ingestion.

The shape agents parse reliably is flat: one # H1, a > blockquote summary, ## sections, and - [Label](absolute-url): description bullets. One level of nesting, every link absolute, every bullet self-describing. ## Optional has defined semantics — skippable when context is tight — and no other section name does. The content type is a genuine fork in the road: text/markdown; charset=utf-8 is the literal answer and most static hosts get it wrong, but no browser renders that type, so a person who follows the link from your robots.txt is handed a download prompt instead of a file. Every widely cited llms.txt in the wild is served as text/plain, and so is this one. The convention is that the content is Markdown, not the header.

• Not a ranking factor. No engine has published a signal that consumes it.
• Not a submission mechanism. Writing a URL into it does not queue a crawl.
• Not a substitute for HTML. A Markdown index of pages a crawler cannot read is still nothing.

import {BASE_URL} from '@/lib/constants';
import {PUBLISHED_SERVICES} from '@/lib/services';

// Pure function of build-time constants — never recompute it per request.
export const dynamic = 'force-static';

/** Hand-written one-liners. A slug is not a description. */
const SUMMARY: Record<string, string> = {
  'nextjs-development':
    'Next.js App Router builds focused on speed, indexability and modern UI.'
  // …one line per published service.
};

function body(): string {
  const services = PUBLISHED_SERVICES.map(
    (s) =>
      `- [${s.slug}](${BASE_URL}/services/${s.slug}): ${SUMMARY[s.slug] ?? ''}`
  ).join('\n');

  return `# Aviral Gupta — Web Developer

> Senior full-stack developer based in Dubai, UAE, nine years across Next.js,
> React, TypeScript and Node, with technical SEO, analytics and AI-crawler
> readiness engineered into the code rather than bolted on after launch.

## Services

${services}

## Facts

- Last reviewed: 22.08.2026
- Canonical origin: ${BASE_URL}. Every other host redirects here.
`;
}

export function GET() {
  return new Response(body(), {
    // text/plain, though the body is Markdown: no browser renders
    // text/markdown, so a reader who clicks the link gets a download.
    headers: {'Content-Type': 'text/plain; charset=utf-8'}
  });
}

Two details do most of the work there. Every URL is built from one origin constant, so no link redirects — an agent that fetches your map should not burn a hop on each entry. And it carries a date: an undated file reads as undatable, which to a retrieval layer that weights freshness is the same as stale.

Do AI crawlers execute JavaScript?

No, and this is the most expensive fact in the subject. As of mid-2026 none of the major AI crawlers execute JavaScript — not GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot or Meta-ExternalAgent. They fetch raw HTML, extract text, and move on: no render wait, no second attempt. Googlebot is the only major crawler with full rendering, which is why a site can rank in Google and be invisible in every answer engine at once.

Instrumented crawler studies published in 2026 (SearchOptimo, 2026) found GPTBot downloading JavaScript in roughly 11.5% of requests and ClaudeBot in roughly 23.8% — and neither executing any of it. Downloading a bundle is not running it. Anything that exists only after hydration does not exist for AI retrieval.

The failure is rarely a whole page. It is one component, and the pattern is conditional mounting: {isOpen ? <motion.div>…</motion.div> : null} in an accordion. I found exactly this on my own homepage — five of six FAQ answers existed only inside <script> tags, because AnimatePresence never mounts a closed panel. One sentence prevents it: always render the content, animate its presentation. Native <details>/<summary> does that for free, needs no client component, and gives you keyboard semantics as a bonus.

Verify it the way a crawler sees it. Strip the scripts, strip the tags, and check your key sentences survive as prose.

# Raw HTML, fetched as GPTBot. No rendering, no hydration.
curl -sS -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://guptaaviral.com/ > raw.html

# Strip <script>/<style>, strip tags, collapse whitespace, then look for your claim.
python -c "import re; h=open('raw.html',encoding='utf-8').read(); \
  b=re.sub(r'<(script|style).*?</\1>','',h,flags=re.S); \
  print(re.sub(r'\s+',' ',re.sub(r'<[^>]+>',' ',b)))" | grep -c "your key sentence"

Which AI user agents should robots.txt name, and what does blocking each cost?

Name every one of them — in comments — and keep exactly one directive group. The reason is RFC 9309 §2.2.1: a crawler obeys the single most specific group matching its user-agent and ignores every other group, including the wildcard. A User-agent: GPTBot block with Allow: / under it therefore looks welcoming and is a trap. The day you add Disallow: /drafts/ to the wildcard group, that rule silently will not apply to any bot you named — precisely the bots you care most about.

# guptaaviral.com — open to search engines and AI answer engines.
# Last reviewed: 2026-05-27
#
# ONE group by design. RFC 9309 §2.2.1: a crawler obeys only the most specific
# matching group and ignores all others, so per-bot groups would shadow this
# one and any Disallow added here would not apply to the named bots.
#
# Absence of a Disallow is consent. Everything below is welcome:
#
#   Live retrieval (citations)  OAI-SearchBot, Claude-SearchBot, PerplexityBot,
#                               Bingbot, Googlebot, DuckAssistBot
#   User-triggered fetch        ChatGPT-User, Claude-User, Perplexity-User,
#                               MistralAI-User
#   Training corpora            GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent,
#                               Amazonbot
#   Opt-out tokens, not bots    Google-Extended, Applebot-Extended
#                               (left unset — training permitted)
#
# Structured summary for agents: /llms.txt

User-agent: *
Allow: /

Sitemap: https://guptaaviral.com/sitemap.xml

The distinction governing every decision below is live retrieval versus training. A live-retrieval bot fetches a page because a user asked a question right now, and what it returns can carry a link back to you. A training crawler donates your text to a future model with no attribution and no click. They are separate user agents: blocking GPTBot does not block OAI-SearchBot, and blocking ClaudeBot does not block Claude-SearchBot. Most people who think they have opted out of AI have blocked one and left the other running.

Two footnotes. Google-Extended and Applebot-Extended are not crawlers — they are robots.txt tokens governing how content Googlebot and Applebot already fetched may be used for training, so Allow: / for them is a semantic no-op. And robots.txt is a request, not a fence: Cloudflare delisted Perplexity as a verified bot in August 2025 after documenting it rotating user agents and IPs to evade no-crawl directives. If your policy is "everyone welcome", that is moot. If it ever becomes "some blocked", you need edge enforcement, not a text file.

Why do redirect hops cost more with AI crawlers than with Googlebot?

Because the live-retrieval bots give up far sooner. Redirect testing published in April 2026 found that OAI-SearchBot and Claude-SearchBot abandon a URL after about three hops, while GPTBot, ClaudeBot and PerplexityBot tolerate five and Googlebot follows up to ten (captaindns, 2026). The bots with the shortest patience are exactly the two that produce citations.

Three hops sounds generous until you count what a normal site spends them on.

• An inbound link on http:// — one hop to HTTPS.
• An apex/www split where the canonical names the host that redirects — one hop, on every URL.
• A trailing-slash normalisation rule — one hop.
• Locale detection that 307s off the canonical path on Accept-Language — one hop, with a different destination per request.

That is four, and the citation crawler left at three. I had this on my own domain: BASE_URL named the apex while production served www, so every canonical tag, every hreflang alternate, every sitemap <loc>, every JSON-LD @id and all twelve links in llms.txt pointed at a host answering 308. Nothing was broken in a browser. Everything was one wasted hop for a crawler. The fix was one constant — which is the shape of this problem generally, and why seo-safe migrations live or die on whether every absolute URL derives from one origin that answers 200.

What structured data actually helps an AI answer engine?

None of it is required, and there is no special AI schema. Google states this plainly in its 2026 AI optimisation guide, and anyone selling "AI schema markup" is selling folklore.

What it does do is disambiguate entities. Before an engine decides whether to cite a page, it decides who and what the page is about. Correct markup turns "some developer in Dubai" into a resolvable node with an identifier, a job title, a location and external profiles that agree with each other. That is the step before citation, not a substitute for it.

Four rules carry almost all the value. Emit one <script type="application/ld+json"> per page containing one @graph, not several disconnected objects. Give every node a stable @id — a node without one cannot be referenced and is discarded at merge time. Make links reciprocal, so Person.worksFor mirrors ProfessionalService.founder. And give every page a WebPage node carrying datePublished and dateModified wired to real content changes rather than build time; a dateModified that moves on every deploy is noise and gets discounted.

import type {BlogPost} from '@/content/types';
import {BASE_URL} from '@/lib/constants';

export const PERSON_ID = `${BASE_URL}/#person`;
export const WEBSITE_ID = `${BASE_URL}/#website`;

export function postGraph(post: BlogPost, url: string) {
  return {
    '@context': 'https://schema.org',
    '@graph': [
      {
        '@type': 'Person',
        '@id': PERSON_ID,
        name: 'Aviral Gupta',
        jobTitle: 'Senior Full-Stack Web Developer',
        address: {'@type': 'PostalAddress', addressLocality: 'Dubai', addressCountry: 'AE'},
        // The primary mechanism for deciding this node and the real person
        // are the same entity. Aim for 5–8 claimed, consistent profiles.
        sameAs: ['https://www.linkedin.com/in/g-aviral']
      },
      {
        '@type': 'WebPage',
        '@id': `${url}#webpage`,
        url,
        isPartOf: {'@id': WEBSITE_ID},
        about: {'@id': PERSON_ID},
        inLanguage: 'en',
        datePublished: post.published,
        dateModified: post.updated ?? post.published
      },
      {
        '@type': 'BlogPosting',
        '@id': `${url}#post`,
        headline: post.title,
        description: post.standfirst,
        author: {'@id': PERSON_ID},
        mainEntityOfPage: {'@id': `${url}#webpage`},
        datePublished: post.published
      }
    ]
  };
}

Two things to skip. Do not emit Review or AggregateRating about yourself on your own site — Google treats that as self-serving and the page becomes ineligible for review rich results, whether the markup is yours or comes from an embedded widget. Publish testimonials as prose with named attribution instead; a model needs no markup to quote a named person. And do not add a SearchAction if you have no site search: a potentialAction pointing at an endpoint that does not exist is a fabricated capability, and an agent that tries it gets an error.

The cheapest unfinished item on most sites is sameAs. One LinkedIn URL is a weak claim. Five to eight claimed, consistent profiles is what lets an engine merge your node with the entity it already knows — an afternoon of work and nothing recurring.

How do you write a passage a model can lift?

Write so that every paragraph survives being torn out of the page alone. Retrieval splits a document into chunks, embeds each one, and pulls the chunk closest to the question. A chunk that needs its neighbours to make sense cannot be cited, no matter how good the article is as a whole. That single mechanism explains almost every content rule worth following.

• Answer first. The direct answer in the first sentence after the heading, then evidence, then nuance. Not the other way round.
• No orphan pronouns. "As mentioned above", "this service", "he" — all unresolvable once the chunk is extracted. Repeat the entity name once per section.
• Write definition sentences. One flat sentence of the form X is a Y that Z for every term you want to own. Complete propositions get extracted disproportionately.
• Number plus attribution. A model quotes "cut LCP from 4.2s to 1.1s (CrUX, 75th percentile mobile, March 2026)" far more readily than the bare number.
• Use real tables. Genuine <table> markup survives chunking as a coherent unit. Comparisons rendered as styled divs do not.
• Phrase headings as the question typed. "How much does a Next.js website cost in Dubai?" beats "Our Pricing Philosophy".
• 150–300 words under each heading. Coherent sections, not fragments — Google is explicit that content need not be broken into tiny pieces.

Then date it visibly. Freshness is weighted heavily: 2026 citation research from Digital Applied found roughly half of AI citations go to content published in the last thirteen weeks, and a page loses about half its citation potential within twelve months. A page with no visible date and no dateModified is published pre-decayed. It is also the argument for having an editorial surface at all — a handful of service pages gives retrieval almost nothing to match a long-tail question against, which is the gap Next.js development should close by construction.

One note on FAQ markup. Google deprecated FAQ rich results on 7 May 2026, so the schema no longer buys SERP real estate. The type is not deprecated and unused structured data does not harm Search; its remaining value is machine-readable Q&A for extraction — worth keeping, provided the answers are also in the visible HTML rather than only in the JSON-LD.

How do you verify all this, and where should the effort go?

Start by proving the crawlers get a 200 at all, because this fails more often than anyone expects. 2026 crawlability research from Anagram found 17.6% of top sites that explicitly allow GPTBot in robots.txt return a 403 to it in practice — a bot-management rule, a rate limit or a WAF challenge overriding the policy the site thinks it has. Nothing in your application logs will tell you. One command will.

for ua in \
  "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)" \
  "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"; do
  curl -sS -o /dev/null -A "$ua" \
    -w "%{http_code}  %{size_download} bytes  %{num_redirects} hops\n" \
    https://guptaaviral.com/
done

A smaller body than a browser fetch means you are being served a challenge page. A non-zero hop count on your canonical host means you are spending a live-retrieval bot's three-hop budget before it reaches your content. Then three checks monthly: Bing Webmaster Tools' AI Performance report, which since February 2026 is the only first-party AI-citation data any engine publishes; a fixed prompt set run in logged-out sessions across ChatGPT, Claude, Perplexity and Copilot, recorded as cited-with-link, mentioned-without-link or absent; and pasting a URL into an assistant with "summarise everything on this page" — anything the summary omits is content your markup is hiding.

None of this is exotic. It is a rendering audit, one origin constant, a robots.txt rewrite, a JSON-LD refactor and an editing standard — the same work I do inside a Next.js build or a technical SEO engagement, because it is the same codebase and the same afternoon. If you want to know which of these your site fails today, get in touch: I will run the checks above against your domain in an introductory project call.