What Actually Gets Your Site Crawled by AI? Our Two-Week Data, and the Mistake We Retired

You spend a week adding AI discovery endpoints, refreshing every date on the site, and wiring up structured data. Your dashboard then shows a 778-request day — a win, right? Then you look closer and half the traffic is a scanner probing /.env.production.

This is the honest two-week readout from GEO Encyclopedia: the daily crawl numbers, which of our own changes produced real AI crawler visits, which produced nothing, and how we learned to tell a real spike from an attack wearing a crawler's name.

---

What Did Two Weeks of Crawl Logs Show?

Content additions produced the real crawl spikes; infrastructure endpoints produced zero requests. Almost every daily spike we saw was either a new article launch or a scanner attack — rarely an infrastructure win.

We log every request through our Cloudflare Worker into a D1 database, tagging crawlers by user agent and recording path, status and page type. Here is the day-by-day picture since logging went live:

Date	Total	Bot	Notable events
08-04	303	90	Worker logging goes live; baseline set
08-05	59	18	Origin switched to raw fetch; CSP sandbox fix
08-08	281	122	GPTBot 59 requests; freshness refresh shipped
08-09	172	28	Community module + two JSON-LD articles launch
08-10	459	97	Content template articles; security headers; checker script
08-11	778	559	⚠️ Scanner spike — ChatGPT-User and Amazonbot probing sensitive paths
08-12	356	70	New article crawled 10× on launch day; homepage FAQPage
08-13	502	182	GPTBot 95; includes a spoofed-GPTBot attack blocked by our Worker
08-14	222	148	Entity-density definition moved above the fold
08-15	187	127	Amazonbot 82 — 67 of them 404s from its own URL-parsing bug

Two things jump out. First, the biggest "success" day — 778 requests — was mostly noise. Second, our single biggest real day for legitimate crawlers was not infrastructure work at all; it was the day a new article went live and GPTBot and ClaudeBot visited that exact URL 10 times.

Which Changes Actually Drove Real Crawls?

We grouped our two weeks of commits into categories and measured each against the logs. The pattern was consistent and repeatable:

Change	Result	Evidence
New articles	Immediate, repeated crawls	New pages hit 6-10 times on launch day, revisited for days after
Definition-first layout (answer before the fold)	Higher share of crawled content pages	The entity-density page became one of the most-crawled content URLs
FAQPage markup on metrics/case pages	New visits to those pages	ClaudeBot and OAI-SearchBot hit the six pages within a day of markup
Freshness alignment (dateModified ↔ lastmod)	Correlated with crawler activity	Peak legitimate crawler days followed freshness passes
/ai/summary.json, /.well-known/ai.txt	Zero requests	No crawler ever requested any AI-facing endpoint in the window
Security headers (CSP)	No crawl impact	Defensive only; never expected a positive crawl effect

The lesson we took away: crawlers find content through llms.txt, robots.txt and sitemap.xml — not through custom AI endpoints. GPTBot and ClaudeBot visited robots.txt and sitemap.xml almost every single day. They never requested a single AI JSON endpoint.

How Do You Tell a Real Spike from a Scanner Attack?

The 08-11 spike taught us a rule we now apply to every "win": look at the paths, not the totals. A real crawl requests your real URLs. A scanner requests things you never wrote.

• ChatGPT-User on 08-11 (117 requests): probing /debug, /.env.production, /api/ping, /telescope/requests, /web/.env.
• Amazonbot on 08-11 (148 requests): probing /wp/.env, /admin/config, /var/www/html/.env, /fetch.
• Spoofed GPTBot on 08-13 (94 requests): concentrated in a single early-morning burst scanning for .env and .git, blocked by our Worker's sensitive-path filter.

Scanner bots borrow well-known crawler names because many firewalls whitelist those names. Your log then shows "ChatGPT is reading my site a lot!" — when the assistant's real crawler never touched you. Filter by status and path before you celebrate.

Did the AI Discovery Endpoints Fail, or Just Not Get Used Yet?

We added /ai/summary.json, /ai/faq.json, /.well-known/ai.txt and related files during the first week — and in two weeks of logs, zero crawlers requested them. That is not proof they are useless; it is proof they are not the current entry point.

Today's answer engines discover content the classical way: robots.txt, sitemap.xml, then the content itself. The AI-facing endpoints are a bet on where tooling is heading, not where it is today. We keep them maintained (they help audit tools like GeoReady and future agents), but we stopped counting on them for crawl growth and moved that effort to content.

What Should a Site Owner Do With This Data?

Prioritize in this order: publish new answerable content, keep robots and sitemap healthy, keep dates fresh, add FAQPage markup — and treat infrastructure endpoints as a low-cost hedge, not a growth driver.

1. Publish new content first. It is the only change in our data that produced immediate, repeated crawler visits.
2. Keep robots.txt and sitemap.xml complete and current. They are the actual entry points crawlers use.
3. Align freshness signals. Match dateModified to sitemap lastmod after every edit.
4. Add FAQPage markup to metrics and case pages. It turns ordinary pages into directly quotable answer surfaces.
5. Keep AI endpoints but don't depend on them. They cost little and may matter later.
6. Filter attacks before reading your dashboard. Separate scanner probes from real crawls in your logging.

Every site is different — our sample is one site, twelve days. But the ordering matches the answer-assetization principle: AI visibility follows content that can be quoted, not configuration that can be probed.

FAQ

Which GEO changes actually increase AI crawler traffic? Publishing new articles drove the biggest real spikes — new pages were crawled 6-10 times on launch day and revisited for days. Freshness alignment and FAQPage markup also correlated with crawler activity. Infrastructure files like /ai/summary.json were never requested by any crawler.

How do you tell a real crawl spike from a scanner attack? Check the requested paths, not just the count. A genuine crawl hits your real URLs; a scanner probes /.env.production, /debug, /admin/config and similar sensitive paths using a borrowed crawler user agent.

Do AI discovery endpoints like /ai/summary.json get crawled? Not in our two-week window. GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot visited robots.txt and sitemap.xml almost every day but never requested any AI-facing endpoint.

What should a site owner prioritize for AI visibility: new content or technical fixes? New content first, then structured data. Content additions produced immediate, repeated crawler visits in our data, while infrastructure endpoints produced zero.