AI crawlers are already reading your WordPress. Here's how to see them

I was reading through our own access log this morning — the raw one, the thing most people never open — and there they were, mixed in with the normal traffic: ClaudeBot, a GPTBot request, a couple of others. Nobody told them to come. They just do.

Here’s the part that stuck with me. Every single one of those AI-assistant requests went to the same place: the sitemap. Not the homepage, not a blog post. /sitemap_index.xml, over and over. That’s how an AI reads a site now — it grabs your map first and works out where everything is, before it reads a word of your content.

If you’ve never looked, you don’t know whether ChatGPT, Claude, and Perplexity can see your site at all. And right now, being readable by them is turning into the same kind of table-stakes that being readable by Google was fifteen years ago.

Who’s actually knocking

These are the user-agents worth knowing. They show up as plain text in your access log:

Crawler	Who it feeds
GPTBot, OAI-SearchBot, ChatGPT-User	OpenAI / ChatGPT
ClaudeBot, Claude-User, anthropic-ai	Anthropic / Claude
PerplexityBot, Perplexity-User	Perplexity
Google-Extended	Google’s AI training (separate from Googlebot)
CCBot	Common Crawl (feeds a lot of models)

A few of them split hairs: GPTBot is training, OAI-SearchBot is for the search product, ChatGPT-User is a live fetch when someone asks ChatGPT about your page. You can allow or block each one separately.

See them yourself in two minutes

If you can reach your access log, this is all it takes:

grep -iE "GPTBot|ClaudeBot|PerplexityBot|Google-Extended|CCBot|anthropic" access.log

Watch where they go. In our logs it was the sitemap every time, which tells you something practical: if your sitemap is broken or incomplete, the AI’s picture of your site is broken too. That’s a cheaper thing to get right than most people realize.

Should you block them? Usually not

The instinct, especially after you see bots eating bandwidth, is to block everything. For AI crawlers that’s usually a mistake. Blocking GPTBot and ClaudeBot doesn’t protect much — it just makes you invisible in the place people increasingly ask their questions. You wouldn’t Disallow Googlebot to save a few requests; this is the same bet, one product cycle later.

Two levers actually matter:

• robots.txt — the on/off switch per crawler. If you want the AI visibility, explicitly Allow them here; don’t leave it ambiguous.
• llms.txt — a short file at your root that tells an AI, in plain language, what your site is and what it should cite you for. It’s the difference between an AI guessing and an AI quoting you the way you’d want.

Where this fits for us

Site Doctor checks whether these crawlers can actually reach and parse your pages — part of every scan, read-only. SysWP Radar goes further and classifies your traffic so you can watch AI crawlers as their own category, not buried in “bots.” But you don’t need either to start. Open your access log today and grep for the names above. Seeing who’s already reading you tends to change how you think about the next thing you publish.