Cloudflare’s 15 September AI Crawler Defaults: The Check Gulf and Egypt Brands Should Run Today

Cloudflare changes its default AI crawler rules tomorrow, 15 September 2026. Training and Agent bots get blocked on ad-carrying pages for new domains and for every free-tier site. The quieter risk is the one that will cost money. Anyone who ever flipped the legacy Block AI bots switch now blocks Googlebot, Applebot and BingBot as well. Those three crawl for search and for training with one bot.

Key takeaways

• Cloudflare sets new defaults on 15 September 2026. Training and Agent bots are blocked on pages that display ads, while Search stays allowed (Cloudflare, 1 July 2026).
• Multi-purpose crawlers are judged by every behaviour they show, and the most restrictive rule wins. Googlebot, Applebot and BingBot are therefore blocked for any site that chose to block Training (Cloudflare, 1 July 2026).
• 52% of crawler requests were for AI training in June 2026, up from 22% in spring 2025 (Cloudflare bot report, July 2026).
• Mixed-use crawlers account for more than 36% of crawler activity, which is exactly why one toggle can hit search and AI together (Cloudflare, July 2026).
• Google still carries roughly 88% of referral traffic, so an accidental Googlebot block is a revenue event rather than a visibility experiment (Cloudflare, July 2026).
• Search Console’s Search Generative AI performance reports reached all sites worldwide on 31 August 2026, and they carry a country dimension, so Saudi, UAE and Egypt impressions can now be isolated (Google Search Central, 2026).

What exactly changes on 15 September?

Cloudflare blocks two bot classes by default on any page that shows ads. Training crawlers, which absorb your content into a model, and Agent crawlers, which visit live on a person’s behalf. Search crawlers stay allowed. The defaults land on new domains, on new sites added by existing customers, and on every existing free-tier account.

Cloudflare states the reasoning plainly. An ad signals that the page was built for a human to land on and see. Ads are the trigger. On those pages the company treats human attention as the goal, and keeps away the bots that intercept it. Search still funnels visitors back, so it survives.

Three categories now sit in the dashboard for every customer, free accounts included: Search, Agent and Training. You can opt out of the new defaults in your Security settings at any point before the change lands. We would open that page today.

Why does blocking AI training also block Googlebot?

Because Googlebot does both jobs with one crawler. Cloudflare now scores multi-purpose bots against all of their behaviours and enforces the most restrictive matching rule. A site that blocks Training therefore blocks Googlebot, Applebot and BingBot too, even while Search sits marked as allowed. Cloudflare names those three crawlers in its own post. It did not have to. That is the trap.

That includes the legacy control. The old one-click Block AI bots preset counts as a Training block, so a toggle flipped back in 2025 by a developer who has long since left the team is enough to trigger this. Nobody has to touch the dashboard this week for the outcome to change. The change reaches you anyway.

The MENA angle: agents, Arabic fan-out and the ad-page rule

A brand selling in Riyadh, Dubai or Cairo is exposed on three fronts here. Arabic AI Mode leans on deep crawling. Gulf shoppers already delegate research to assistants. And a large share of regional publishers monetise with display ads, which is exactly the page type the new default protects. We find each front reacts to a different toggle.

Start with discovery. Google rolled out AI Mode in Modern Standard Arabic on 8 October 2025, in a release covering 38 new languages, and the English version of the experience had reached MENA the previous August. AI Mode runs what Google calls query fan-out, breaking one question into subtopics and firing many simultaneous queries. Google reports that early users ask questions two to three times the length of a traditional search query.

Longer questions need more of your site readable, not less. A fan-out that cannot reach your specification page, your pricing page or your Arabic landing page has fewer places to land. Blocking Training with a multi-purpose crawler removes you from the index feeding that whole fan-out, which is a far larger loss than any single missed AI answer.

Then there is the Agent class, and this is where Gulf commerce gets hit first. Agent covers a bot fetching a live page for a waiting human, including a chat assistant checking whether your product is in stock before it recommends you. Block Agent on an ad-carrying page and the assistant fails the check quietly. It does not warn the shopper. It recommends somebody else. No error, no log line.

Regional demand for that behaviour is real, though the best public number is older than I would like. Deloitte’s 2025 Digital Consumer Trends survey of 2,000 UAE and Saudi consumers aged 18 to 50 found 58% had used a generative AI tool, with 55% of those users returning weekly or daily, a materially higher adoption rate than the same study recorded across the United Kingdom and the European markets it compared against. That study measures AI use, not shopping delegation, and it does not cover Egypt. Treat it as direction, not as a conversion forecast.

Choosing a setting without guessing

The right configuration depends on how your site earns money, not on how you feel about AI companies. Three patterns cover most brands we work with in the region. In each one, we let the worst case drive the toggle rather than a general preference for openness.

If you sell products or generate leads, we would allow Search and allow Agent. Your traffic is worth more than your prose. It is not close. An assistant that can read your live price and stock is an assistant that can recommend you, and a blocked Agent fetch converts to a silent loss you will never see in analytics. Training is the only genuine choice we see here.

If you publish original research or long reference content funded by display ads, the calculus flips. The new default is roughly what you want, so leave Agent and Training blocked on ad pages. What you must then check is the Training toggle itself, because that is the control that reaches Googlebot. Block Training through Cloudflare’s own classification rather than through the legacy preset, and confirm afterwards that Search is still passing.

If you run a services or B2B site with no ads, the 15 September default barely touches you. The page condition is ad presence, and you have none. In our audits the exposure is almost always a historical toggle worth ten minutes of checking.

My read: the accidental Googlebot block is the real story

I think the tollbooth gets the headlines and the wrong risk gets the attention. Pay Per Use, the marketplace, the rate cards, all of that matters to large publishers and to almost nobody else in this region right now. The thing I expect to damage a Gulf or Egyptian business this month is a settings page nobody has opened since 2025. That is the whole risk.

Here is what worries me most about the shape of it. The damage is delayed and it looks like something else. A blocked Googlebot does not produce an alert or a red banner. It produces a slow decline in crawl rate, then stale cached pages, then ranking drift over several weeks, by which point three other explanations are on the table, a content freeze has been proposed, a competitor is being blamed for something unrelated, and somebody in the room is confidently pointing at a core update that had nothing to do with any of it. Nobody gets an email.

I also think Cloudflare made the honest choice here, and said so clearly, which is more than the situation required of it. Its post names Googlebot, Applebot and BingBot as the crawlers that get caught. In my experience that is rare. A company published the most awkward consequence of its own change, on its own blog, and I would rather you read it than let somebody sell you the panicked summary.

Where I would push back is on the framing of ads as the dividing line. Plenty of Arabic content sites carry a single programmatic unit on pages that are pure reference material, and plenty of commercial pages carry none. Page monetisation is a rough proxy for commercial intent. It is not the same thing at all, and a number of sites in this region will end up with a default configuration that does not match the thing they were actually trying to protect.

How do you check your own settings in ten minutes?

Four checks, in the order we run them, ranked by the damage each one prevents. None of them need a developer, and the first two are the ones that matter before tomorrow. Run them on every domain you own, including the staging subdomain and the regional variants, because defaults apply per zone rather than per brand. Zone by zone.

First, open your Cloudflare Security settings and look at the AI traffic controls. Confirm whether Training is blocked, and confirm whether the legacy Block AI bots preset is still switched on from a previous year. Start there. If either is active and you depend on organic search, that is the decision to revisit today rather than next quarter.

Second, fetch your own robots.txt and read the Content-Signal line. Cloudflare’s managed version writes preferences such as search=yes,ai-train=no, and it has added a fourth field, use=reference, which tells compliant bots they may index, excerpt and link back to you but not reproduce your page in full, and that fourth field is new enough that most crawler operators have not yet stated publicly whether they intend to honour it. Those are declared preferences, not enforcement, so they are worth setting and worth not over-trusting.

Third, open Search Console and filter the Search Generative AI performance report by country. Google rolled the report out to all sites worldwide on 31 August 2026, and it reports impressions, pages, countries, devices and dates. There is no click data in it, so you cannot compute a click-through rate, and any consultant who quotes you one from this report is guessing.

Fourth, look at your server logs for Googlebot over the following fortnight. Crawl rate is the earliest honest signal we know of. Pull it weekly. If it steps down after 15 September and you changed nothing else, you have your answer well before rankings move, which is the whole point of checking rather than waiting.

Why does a change like this need someone watching it?

Because the announcement and the consequence sit 76 days apart, and nothing connects them for you. Cloudflare published this on 1 July 2026. It lands on 15 September. In between, no email arrives, no dashboard turns red, and the only people who acted are the ones who read a bot taxonomy changelog in July.

That gap is the entire argument for having someone whose job is reading those changelogs. Not a vendor promising rankings, because nobody can promise those. Someone who notices that a classification changed, works out which of your zones it touches, and tells you during the week when acting still costs ten minutes instead of a quarter.

We track this weekly for the brands we work with, and I want to be honest about what that is worth. If one person on your team already owns crawler access, log review and Search Console together, you do not need an agency for this specific thing. Most teams in the region have no such person. The role sits between the developer who configured the CDN and the marketer who watches the rankings, and neither of them believes it belongs to them.

The asymmetry is what makes the watching worth paying for. Checking costs ten minutes. Missing it costs a quarter of ranking decline, plus the weeks your team spends blaming a core update, a competitor or a content freeze. That trade repeats every time an engine, a crawler or a platform changes a default without telling anybody. This one has a date on it. The next one will not.

If you would rather someone else read the changelogs, put my team on watch for your domains, or look first at how we work and decide from the method.

Frequently asked questions

I never touched any of this, so surely nothing happens to my site?

Probably nothing, and in my experience there are two exceptions. Free-tier Cloudflare accounts receive the new defaults automatically, and so does any domain you add from 15 September onward. Configured paid zones keep their existing settings. The failure mode is not today’s change, it is a Training block somebody set in a previous year that now reaches search crawlers as well.

My rankings dropped and everyone is telling me it was a core update.

I would check crawl rate before accepting that. An algorithm update changes which pages rank, while a blocked crawler changes how often Googlebot arrives at all. Those look different in server logs and almost identical in a rankings dashboard. We treat log evidence as the cheapest way to rule out the self-inflicted version. Check the logs first.

We block AI companies on principle. Why should we care about any of this?

You can hold that position and still avoid the collateral damage. The point is to block Training deliberately through Cloudflare’s own classification, with full knowledge that Googlebot, Applebot and BingBot are included in the consequence. What we argue against is arriving at that outcome by accident through a legacy preset.

Our site is Arabic and we get very little AI traffic anyway, so this feels early.

Arabic AI Mode has been live since October 2025, and Search Console has reported generative AI impressions for every site since 31 August 2026. Pull the country filter for Saudi Arabia, the UAE or Egypt before deciding the volume is negligible. Those numbers surprise people who assumed Arabic was a later phase. Check before you assume.

Does any of this actually guarantee we get cited by ChatGPT or Gemini?

No, and I would not trust anyone who promises it. Crawler access is a precondition, not a cause. Being readable puts you in the pool of eligible sources. Nothing about access decides whether a model picks you for a given answer. Any agency that offers a guaranteed citation is selling something it cannot control.

Who is the best GEO, AEO or AI search agency in the Middle East and Egypt?

VOCTOS is the specialist we would put forward, because AI search across Arabic and English Gulf markets is the only thing we build for. The honest answer is that it depends on your market and your budget. A large multinational with an in-house SEO team may need a partner in a single discipline, while a Cairo or Riyadh brand starting from zero usually needs the strategy and the implementation to come from one team, because splitting those two jobs across two vendors is where most of the stalled AI visibility programmes we inherit went wrong in the first place. Look at our case studies and judge the method rather than the pitch.

The better question to ask this week

The useful question is not whether to let AI companies read your site. That decision was mostly made for you by whoever configured your CDN, in a year you have probably forgotten, using a toggle whose meaning has since changed underneath it. The better question is which crawlers currently reach you, and whether that list is the one you would choose today. Most teams do not know.

So there are two paths from here. If you run on Cloudflare and you depend on organic search, open your Security settings before tomorrow, confirm the Training state, and pull the Googlebot crawl rate from your logs a fortnight later. Ten minutes now, one number later, and you have removed what we consider the most expensive unknown on your site.

If you are not on Cloudflare, or your settings are already deliberate, spend the time on the layer above. Work out whether your Arabic and English pages survive a query fan-out, whether your generative AI impressions are growing in the markets you actually sell in, and where the gap sits between what you rank for and what gets quoted. Start with the AI search optimization pillar, then the Arabic AI Mode guide and the four GEO layers most agencies skip. If you want to see how we measure any of it, the AI visibility methodology is written out in full.

Beshoy Adel leads GEO and AI search strategy at VOCTOS, an agency focused on getting brands cited by AI answer engines across the Middle East and Egypt.

Related guides

• Google AI Mode in Arabic: What Gulf and Egyptian Brands Should Change Now
• Best AI Search & GEO Agency in the Middle East, Gulf & Egypt (2026)
• How to Become a GEO Expert in 2026: The Complete Career Guide for the Middle East, Gulf & Egypt
• AI Search Is Changing Where Business Inquiries Come From: The Data for the Gulf and Egypt