What GEO Work Actually Moves AI Crawlers? Our Data, and a New-Site Launch Checklist

You are about to launch a new site and the advice keeps multiplying: add /ai/summary.json, wire up llms.txt, sprinkle FAQPage markup, build a discovery layer. But which of these does an AI crawler actually care about?

We run a Cloudflare Worker that logs every request to geo010.com and tags it by crawler, so we could answer that question with data instead of opinion. This article attributes two weeks of AI crawler traffic to nine GEO measures, shows which ones changed nothing, and turns the results into a launch checklist for a new site.

---

How Did We Attribute Traffic to Each GEO Measure?

Every request to geo010.com passes through a Worker that records the date, path, status, page type and a normalized crawler name into a D1 database. For each of nine GEO measures we shipped, we then asked the logs three questions: which crawlers hit the relevant paths, how often, and did visits change after the measure shipped?

Two warnings before the numbers. First, a crawler label is only a user agent — attackers spoof them, so we cross-checked paths and status before counting anything. Second, some "visits" were us: we verify deployments with curl, and our own checks show up in the logs. We filtered those out.

Which GEO Measures Actually Drove Crawler Traffic?

Robots.txt and sitemap.xml were the only paths every major AI crawler used. The homepage was the only page they all visited. Everything else showed up in a minority of crawlers — or none at all.

Here is the full attribution table, ranked by how much real crawler traffic each measure produced:

GEO measure	Which crawlers	Evidence (Aug 4-16)	Impact
robots.txt	ClaudeBot, Googlebot, OAI-SearchBot, Bingbot, FacebookBot	ClaudeBot 68, Googlebot 19, OAI-SearchBot 18, Bingbot 12	Highest — nearly daily entry point
sitemap.xml	ClaudeBot, Bingbot, GPTBot	ClaudeBot 68, Bingbot 8, GPTBot 6	Highest — ClaudeBot reads it religiously
Homepage /	GPTBot, PerplexityBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Applebot	GPTBot 17, PerplexityBot 12, OAI-SearchBot 10, ClaudeBot 8	High — visited by every major crawler daily
New content pages	GPTBot (primary), ClaudeBot, PerplexityBot	GPTBot 66 of 83 requests hit content pages	High — GPTBot traverses all pages
Freshness alignment	All (indirect)	Crawler bursts on 08-08 and 08-12 followed freshness passes	Medium — sustains re-visits
llms.txt	GPTBot, Amazonbot, Googlebot	4 total requests, GPTBot 2	Low — only GPTBot treats it as an entry
Definition-first layout	GPTBot, ClaudeBot, Bingbot, others	Entity-density page reached by 7+ crawlers	Low-medium — quality, not quantity
FAQPage markup	None (AI)	0 visits from major AI crawlers after shipping	None (crawl) — a citation-layer lever
AI discovery endpoints (/ai/, ai.txt, plugins)	None	0 requests in two weeks	None — not used by today's crawlers

The ranking surprised us. The "boring" infrastructure files — robots.txt and sitemap.xml — produced more genuine crawler traffic than every shiny AI feature combined.

Why Do GPTBot and ClaudeBot Crawl Completely Differently?

The two most important AI crawlers could not be more different in behavior, and the difference determines what to build first:

Behavior	GPTBot	ClaudeBot
Reads robots.txt	Rarely (1 request)	Religiously (68 requests)
Reads sitemap.xml	Sometimes (6)	Every day (68)
Visits content pages	Yes — 66 of 83 requests	Rarely (a handful total)
Entry strategy	Traverses the whole site per pass	Follows robots → sitemap, then mostly home

ClaudeBot is a disciplined path-follower. It checks your robots and your sitemap almost daily, but it barely descends into content unless something in that sitemap or homepage directs it. GPTBot is a deep scanner. It skips robots, reads llms.txt occasionally, and walks the entire site, hitting every page roughly once per pass.

The practical consequence: if your robots.txt or sitemap.xml is broken, you hurt the crawler that reads them daily (ClaudeBot). If your content is shallow, you waste the crawler that reads everything (GPTBot).

What Did We Get Wrong in Earlier Analyses?

Honesty check. Earlier write-ups on this site credited FAQPage and new-article launches with immediate crawler spikes. Our per-request attribution showed those were misattributed:

• "A new article got crawled 6-10 times on launch day." Actually most of those hits were browser visitors and our own checks; each AI crawler visited the new page once. New content is crawled once per GPTBot pass, not burst-spiked by the act of publishing.
• "FAQPage pages were visited by ClaudeBot and OAI-SearchBot the same day." Not supported by the logs — the six pages got no visits from major AI crawlers after the markup shipped. Their visitors were browsers, Bingbot and our curl checks.
• "The 08-11 traffic spike was AI growth." It was a scanner attack — ChatGPT-User and Amazonbot probing /.env.production and /telescope/requests. Same for a spoofed-GPTBot burst on 08-13.

None of those measures are useless. They just do their work at a different layer: FAQPage helps a page be quoted once AI is already considering it; llms.txt helps GPTBot orient; discovery endpoints may help future tooling. The mistake was measuring them against crawl counts at all.

What Should a New Site Build First?

If we were launching a new site today, we would build three things before anything else: a correct robots.txt, a sitemap.xml where every URL returns 200, and a server-rendered homepage that explains what the site is — plus a few genuinely useful articles.

1. Make robots.txt correct. Allow the AI crawlers you want, block only what you must (analytics, staging, private pages). A wrong rule that blocks everything is a silent zero.
2. Make sitemap.xml 100% valid. Every URL in it must return 200. A sitemap full of 404s teaches the daily visitor (ClaudeBot) that your site is broken.
3. Serve real HTML on the homepage. AI crawlers do not run JavaScript. The homepage must contain the answer to "what is this site?" in plain text, with links to your best pages.
4. Publish 3-5 real articles before launch day. GPTBot traverses everything; give it pages worth reading. Five solid pages beat fifty shells.

That is the entire pre-launch crawl work. It took us the least effort of all nine measures, and it produced the most crawler traffic.

What Should a New Site NOT Build First?

Skip these until the foundations above exist. Our logs show they contribute nothing to crawl, however good they look on a roadmap:

• AI discovery endpoints (/ai/summary.json, /.well-known/ai.txt, ai-plugin.json, openapi.json) — zero crawler requests in two weeks. Audit tools like them; crawlers ignore them.
• FAQPage markup on every page — no crawl effect in our data. Add it later, on pages that are already candidates for citation.
• A monitoring dashboard — start with your server logs or platform analytics for the first week; you only need to know whether crawlers are arriving and what they hit.
• Freshness plumbing — worth building, but only once you have content and dates to keep honest.

This does not mean those features are bad. It means they are phase-two work, and building them before the crawl foundation is optimizing a pipeline that is not flowing yet.

FAQ

What should a brand-new site build first for AI crawlers? A correct robots.txt, a sitemap.xml where every URL returns 200, and a server-rendered homepage with a one-line statement of what the site is plus a few real articles. Those were the only entry points every AI crawler used in our logs.

Do AI discovery endpoints like /ai/summary.json help AI crawlers? Not in our two weeks of logs. GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot visited robots.txt and sitemap.xml almost every day but never requested any AI-facing endpoint.

Why do GPTBot and ClaudeBot behave differently? ClaudeBot follows the classic crawl path — robots.txt and sitemap.xml daily, but barely any content. GPTBot ignores robots, reads llms.txt occasionally, and traverses the entire site — 66 of 83 requests hit content pages.

Does adding FAQPage markup get a page crawled more? In our data, no measurable crawl effect. It improves how a page is quoted once AI is already considering it — it is a citation-layer lever, not a crawl driver.