AI Crawlers Are Eating Your Hosting Plan: The 2026 Bot Traffic Problem

Your traffic graph doubled last month. Your revenue did not. Before you celebrate or panic, check the user agents, because there is a good chance most of that "traffic" was GPTBot, ClaudeBot, Amazonbot and a few dozen crawlers you have never heard of, quietly working through every URL you have ever published.

This has stopped being a curiosity and become a line item. Automated requests made up 57.5% of all web traffic by June 2026 according to Cloudflare Radar, up from 53% across 2025, which was already the first calendar year where machines outnumbered people. AI-driven crawling specifically grew roughly 300% year over year.

Two things follow from that, and most site owners only know about one of them. The first is that bots consume the bandwidth, visits and CPU cycles you are paying for. The second is stranger: some hosts have quietly started blocking AI crawlers on your behalf, which protects your bill and erases you from AI answers at the same time. Both of those are decisions you should be making, not discovering.

The short version

• Bots are now the majority of web requests. On a small site, AI crawlers routinely account for 30% to 50% of raw bandwidth.
• Over half of AI crawler traffic re-fetches pages that have not changed, per Cloudflare's own numbers. You are paying to serve the same article to the same bot repeatedly.
• Kinsta stopped counting bad-bot bandwidth against plan limits in November 2025. WP Engine never counted bots as billable visits, but does rate-limit them at the platform level, which is a different trade-off entirely.
• Unmetered and high-allowance VPS plans are effectively immune to the cost side. Contabo Cloud VPS 10 gives you 4 vCPU, 8 GB RAM and unlimited traffic for 3.60 EUR/mo.
• On 15 September 2026 Cloudflare starts blocking mixed-use AI crawlers by default on ad-bearing pages. If you are on the free tier, that default lands on you whether you asked for it or not.

How much traffic are we actually talking about?

The numbers are worse than the abstraction suggests. In a single 24-hour window, bots hit add-to-cart URLs on WordPress sites 7.67 million times. ClaudeBot alone was responsible for about 3.75 million of those. Add-to-cart is the single most expensive request a WooCommerce site can serve: uncacheable, database-heavy, session-creating. Bots were hammering it millions of times a day for no commercial reason whatsoever.

One crawler in a documented case generated 550 million requests over 30 days against a single property before anyone stopped it. That is not a traffic spike, that is a sustained load test you did not order and cannot invoice anyone for.

The waste is what stings. Cloudflare found that more than half of AI crawler traffic is re-fetching content that has not changed since the last visit. Your 2023 blog post gets pulled down again and again, at your expense, so a model can confirm it still says what it said last week.

Four ways your host makes you pay for it

1. Visit-metered plans

Premium managed WordPress hosting has historically sold capacity in "visits". The definition matters enormously here. WP Engine counts one visit as one unique IP address per day, and explicitly excludes known and suspected bot user agents, plus requests that only touch static assets. Kinsta filters most well-known bot user agents out of its visit counts too, though it acknowledges that bots which do not identify themselves still consume server resources and may slip into the total.

That exclusion is doing a lot of load-bearing work. A crawler that identifies itself politely as GPTBot is free. A scraper rotating residential IPs with a Chrome user agent is indistinguishable from a reader, and it counts.

Provider	Entry plan	Price/mo	Allowance	Storage	Sites
Rocket.net	Starter	$25.00	50 GB bandwidth	10 GB NVMe	1
Kinsta	Single 35k	$29.17	35,000 visits/mo	10 GB	1
Kinsta	Single 20GB	$29.17	20 GB bandwidth	10 GB	1
WP Engine	Startup	$30.00	75 GB bandwidth	10 GB	1
Cloudways	Autonomous Growth	$99.00	150 GB bandwidth	20 GB	1

Look at the Kinsta rows. Same price, two different meters: 35,000 visits or 20 GB of bandwidth. That choice exists precisely because bot traffic broke the visit model. Kinsta announced on 4 November 2025 that bandwidth consumed by unwanted bots and scrapers no longer counts against plan limits, and gave customers the option to pick whichever meter suits their traffic shape. If your audience is small but your content is heavily crawled, visits are the safer unit. If you serve large media to few people, bandwidth is.

Also note how tight these allowances are in absolute terms. WP Engine's Startup plan includes 75 GB of bandwidth for $30/mo. A determined crawl of a media-heavy site can eat that in days. Compare the same $30 against a VPS plan and you are looking at multiple terabytes.

2. "Unlimited" shared hosting

Shared hosts do not meter your bandwidth, so on paper bots are free. In practice they meter something else: CPU seconds, entry processes, I/O operations, inodes. A crawler that requests 40,000 uncached PHP pages in an afternoon does not blow a bandwidth cap, it blows your CPU quota, and the account gets throttled or suspended. The bill arrives as an upgrade prompt rather than an overage line.

Provider	Plan	Intro price	Renewal	Bandwidth	Storage
ChemiCloud	Starter	$2.49/mo	$11.95/mo	Unlimited	20 GB NVMe
NameHero	Starter Cloud	$2.91/mo	$11.64/mo	Unmetered	25 GB NVMe
GreenGeeks	Lite	$2.95/mo	$13.95/mo	Unmetered	25 GB SSD
Hostinger	Premium	$2.99/mo	$10.99/mo	Unlimited	20 GB SSD

We have written before about what "unlimited" hosting really means. Bot traffic is the fastest way to find out for yourself.

3. Metered egress on cloud platforms

This is the model where a crawl storm turns into a genuinely alarming invoice. Platforms that bill per gigabyte of egress charge the same for a bot as for a customer. There is no bot exclusion, because there is no concept of a visit at all, just bytes leaving the network. We covered the mechanics in detail in bandwidth overage and CDN pricing traps, and AI crawlers have made that article age faster than we expected.

4. Nothing, because your plan has headroom

The unglamorous answer to the whole problem is buying a plan where the traffic simply does not matter. Unmetered European VPS providers charge a flat monthly fee and do not care whether the bytes went to a reader in Berlin or a datacenter in Oregon.

Provider	Plan	Price/mo	Bandwidth	CPU	RAM	Storage
Time4VPS	Linux 2	2.88 EUR	4 TB	1 core	2 GB	25 GB SSD
Contabo	Cloud VPS 10	3.60 EUR	Unlimited	4 vCPU	8 GB	75 GB NVMe
Hostwinds	Unmanaged SSD Cloud 1	$4.99	1 TB	1 vCPU	1 GB	30 GB SSD
NameHero	Starter VPS	$5.44	4 TB	-	-	80 GB NVMe
Hetzner	CX23	5.49 EUR	20 TB	2 vCPU	4 GB	40 GB NVMe
Netcup	VPS 500 G12	5.91 EUR	Unlimited	2 vCPU	4 GB	128 GB NVMe
InterServer	1 Slice	$6.00	1 TB	1 core	2 GB	30 GB SSD

Contabo at 3.60 EUR with unlimited traffic and 8 GB of RAM is the single most bot-proof entry on this page. Hetzner's 20 TB allowance on a 5.49 EUR plan is functionally unlimited for any site that is not distributing video. The trade-off is that you now run the server, which is real work. Our managed vs unmanaged VPS breakdown covers whether that trade is worth it for you, and Hetzner vs Netcup settles the European price fight.

The part nobody warned you about: your host may already be blocking AI bots

Here is where it gets genuinely interesting, and where the cost conversation flips into a visibility conversation.

Testing published in April 2026 found that WP Engine enforces platform-level rate limiting on high-impact crawlers. ClaudeBot requests were rate-limited (HTTP 429) 29% of the time, GPTBot 29%, and Amazonbot 51%. WP Engine support confirmed the behaviour, describing it as protection for overall server performance, and stated it cannot be selectively disabled per bot.

Other managed platforms went the other way. Kinsta's CTO said in March 2026 that Kinsta will not block at the platform level. Pressable's documentation states it does not disallow these bots by default. Pantheon states plainly that it does not block identified bot traffic from entering the platform.

Platform	Blocks AI crawlers by default?	Bot traffic billed?
WP Engine	Yes, platform-wide rate limiting on high-impact bots, not disableable per bot	No, known bots excluded from billable visits
Kinsta	No, stated policy against platform-level blocking	No, bad-bot bandwidth excluded since Nov 2025
Pressable	No, per published documentation	Visit-based plans
Pantheon	No, explicitly does not block identified bots	Visit-based plans
Self-managed VPS	Only what you configure	Only if your plan is metered

Why does that matter beyond principle? Because on the same test property, access rate and citation rate tracked each other closely. GoogleBot had roughly 100% access and the site appeared in 37.8% of Google AI Mode responses. PerplexityBot had 100% access and 7.8% citation presence. ClaudeBot, at 57% access, produced a 0.0% citation rate in Claude.

That is one site, one test window, and correlation is not proof. But the direction is not subtle. Where the crawler can read you, the model cites you. Where it cannot, you are absent from an answer surface that is quietly taking a growing share of the queries that used to become clicks.

So the decision is not "bots bad, block them". It is a trade you should make deliberately: pay a little bandwidth to stay in the index that increasingly mediates discovery, or save the bandwidth and accept invisibility. Choosing by accident, because your host set a default you never saw, is the only genuinely wrong answer.

What changes on 15 September 2026

Cloudflare has been escalating this since it became the first major infrastructure provider to block AI crawlers by default. As of 15 September 2026, mixed-use crawlers, the ones that blend search indexing with AI training and agentic browsing, get blocked by default on pages that carry ads. The default applies to new Cloudflare customers, new sites added by existing customers, and every site on the free tier.

If you run an ad-supported site behind Cloudflare's free plan, that switch flips for you automatically. Worth knowing before it happens rather than after your AI referral traffic quietly flatlines.

Cloudflare has also moved from Pay Per Crawl to Pay Per Use, which pays publishers when their content is actually used in an answer rather than merely fetched. Good idea in principle. In practice it is early, partner-limited, and not something a site doing 5,000 visits a month should be building a strategy around.

What to actually do this week

1. Look at your logs before you change anything. Filter by user agent and count requests and bytes for GPTBot, ClaudeBot, PerplexityBot, Amazonbot, Bytespider, CCBot and Meta-ExternalAgent. Until you know your split, everything else is guesswork.
2. Find out whether your host is already blocking. Ask support directly: do you rate-limit or block AI crawlers at the platform level, and can I opt out? On WP Engine the answer is yes and no respectively. Do not assume your robots.txt is the only thing in play.
3. Decide per crawler, not globally. Training crawlers that never send referrals are a different proposition from retrieval crawlers that cite sources with links. Blocking GPTBot and allowing OAI-SearchBot is a coherent position. Blocking everything with a wildcard usually is not.
4. Put caching in front of the problem. A crawler hitting a full page cache costs you bytes and nothing else. The same crawler hitting uncached PHP costs you CPU, database queries and the throttling that follows. This is the single highest-leverage fix on shared hosting.
5. Rate-limit rather than ban. A crawl-delay or an edge rate limit keeps you in the index while capping the damage. A hard 403 removes you from it.
6. Protect the expensive endpoints specifically. Cart, checkout, search, faceted filters, calendar pages. Those are where 7.67 million daily bot hits become someone's downtime.
7. If bots are more than a third of your bandwidth, reprice. Run the numbers through our true cost calculator. Paying $30/mo for 75 GB while crawlers eat 25 GB of it is a worse deal than 5.49 EUR for 20 TB.

Our recommendation, by situation

Small WordPress site, non-technical owner. Stay on shared hosting, turn on full-page caching, and add a crawl-delay. Hostinger Premium at $2.99 intro and $10.99 on renewal is fine until bots become a CPU problem rather than a bandwidth one.

Content site that depends on being found. Do not host anywhere that blocks AI crawlers for you. That points at Kinsta among the premium managed options, and the bandwidth-metered plan rather than the visits one now that bad-bot traffic is excluded. See Kinsta vs WP Engine for the wider comparison.

WooCommerce store. Your cart endpoints are the target. Get proper edge caching, rate-limit uncacheable paths, and pick a host with real CPU headroom rather than a generous bandwidth number. WooCommerce hosting is worth the premium here.

Developer or agency running your own stack. Take the unmetered VPS and stop thinking about it. Contabo at 3.60 EUR or Netcup VPS 500 G12 at 5.91 EUR removes the cost side of the problem entirely, leaving you to make the visibility decision on its own merits. Browse cloud hosting and VPS hosting for the full list.

Bottom line

AI crawlers turned a rounding error into a budget line and a distribution question at the same time. The cost half is solvable with cheap infrastructure: unmetered European VPS plans start at 3.60 EUR and make the whole conversation moot. The visibility half is not solvable with money, only with an explicit decision about who gets to read your content.

The worst position is the common one: on a metered plan, with a host quietly rate-limiting crawlers you never evaluated, paying for bot bandwidth while disappearing from the answers those bots feed. Check your logs, ask your host the direct question, and pick your side on purpose.

Prices in this article come from the HostingSift database and were current in August 2026. Intro prices are promotional; renewal rates are listed where they differ. Third-party statistics are attributed inline and reflect published figures as of mid-2026.