Do Arab news sites block AI crawlers? A robots.txt study of 107 sites, 2023 to 2026

Most leading Arab news sites still let AI crawlers in. On 21 October 2026, 11 of the 98 top news sites we could read in 12 Arab countries (11.2%) blocked OpenAI’s GPTBot in robots.txt. 81 (82.7%) had no rule for any AI crawler. At the end of 2023 the share blocking OpenAI was 3.2%.

Key numbers

• 11 of 98 sites (11.2%) blocked GPTBot on 21 October 2026. With a sample this size, the 95% interval runs from 6.4% to 19.0%.
• 13 of 98 (13.3%) blocked at least one of the 13 AI crawlers we checked. Bytespider was blocked most often (12 sites), OAI-SearchBot least often (4).
• 81 of 98 (82.7%) had no rule that names any AI crawler, and 7 of those served no robots.txt at all.
• 3 of 98 (3.1%) blocked training crawlers while leaving all four search and user-fetch crawlers open. 10 of the 13 blocking sites also shut out at least one search or user-fetch crawler.
• 3 of 95 (3.2%) blocked OpenAI in December 2023 under the Reuters Institute’s definition, when Reuters found 48% across ten other countries. None of those 3 Arab sites named GPTBot.
• 15 of 98 (15.3%) blocked GPTBot in March 2026, the highest monthly share in the archive.
• 31 of 101 sites with archived history blocked at least one AI crawler at some point between August 2023 and October 2026. 18 of them later dropped every AI block.
• 9 of 91 (9.9%) published an llms.txt file on 21 October 2026.

Free to quote with a link to this page. The full dataset, one row per site, is linked at the end.

The study at a glance

Why we ran this

In February 2024 Richard Fletcher of the Reuters Institute counted how many of the most used news sites in ten countries blocked AI crawlers. His answer, 48% for OpenAI by the end of 2023, became the number people quote.

None of those ten countries was Arab, and nobody had run the count for Arabic news. Editors in the region were making a decision with no regional baseline, and brands trying to appear in Arabic AI answers had no idea which outlets those systems could read.

We wrote the question, the sample rules and the measures down on 21 October 2026 and hashed the file before reading a single robots.txt:

Which AI crawlers do the leading news websites of the Arab world allow or block in robots.txt, how does that differ between crawlers that train models and crawlers that fetch pages for AI search answers, and how has it changed since GPTBot was announced in August 2023?

Which AI crawlers do Arab news sites block?

Few sites block any of them. On 21 October 2026, Bytespider, ByteDance’s crawler, was blocked by 12 of 98 sites (12.2%), GPTBot and ClaudeBot by 11 each (11.2%). OAI-SearchBot, which OpenAI uses to find pages for ChatGPT search, was blocked by only 4 (4.1%). The other 85 sites blocked none of the 13.

Blocking comes in bundles. The 13 sites that block anything block between 1 and 13 crawlers each, and 5 of them block all 9 training crawlers. Most of it is written by name: 11 of the 13 sites list the crawlers they want out.

The other two, addustour.com and alayam.com, use a default-deny file. Their “User-agent: *” group says “Disallow: /”, which shuts out every crawler without its own group, and they then let Google’s crawlers in by name.

One publisher carries a lot of weight in this sample. Al Jazeera’s Arabic site, aljazeera.net, made the top 15 in all 12 countries, and its English site, aljazeera.com, in 5. Both block GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Bytespider and cohere-ai, and both leave OAI-SearchBot and Google-Extended open. Because the Arabic site is counted in every country, every country in the table further down has at least one blocker.

Do they block AI training, AI search, or both?

Mostly both, when they block at all. All 13 blocking sites shut out at least one training crawler. 10 of them also shut out at least one crawler that fetches pages for search or for a user’s question. Only 3 sites (3.1%), al-sharq.com, sabq.org and yabiladi.com, block training while leaving all four search and user-fetch crawlers open.

The split matters because these crawlers do different jobs. OpenAI documents GPTBot as its crawler for model training. OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User opens a page when a user asks for it. Blocking one leaves the others free. OpenAI also says that sites which disallow OAI-SearchBot are not shown in ChatGPT search answers, and that robots.txt rules may not apply to ChatGPT-User, because a user starts those visits.

sabq.org shows the selective approach in its plainest form. One group in its file names GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider and Applebot-Extended and disallows everything; the rest of the file is open.

One grouping decision changes this count. We put Amazonbot with the search crawlers before reading any file, following the pre-registration. 4 of the 10 sites that block a search or user-fetch crawler block Amazonbot and nothing else in that group, 3 of them through Cloudflare’s managed file.

Count Amazonbot as a training crawler and the train-only pattern rises from 3 sites to 7. Either way, only 6 sites block any of the three crawlers that serve ChatGPT and Perplexity answers (OAI-SearchBot, ChatGPT-User and PerplexityBot).

What do the files that block nothing say?

Most say nothing about AI. 81 of 98 sites (82.7%) have no rule that names any AI crawler. Of those, 72 serve a robots.txt with no AI rule, 7 serve no robots.txt, and 2 shut out every unlisted crawler without naming one. A missing file counts as “allow everything”, which is how crawlers read it.

6 sites name AI crawlers only to let them in: youm7.com, royanews.tv, babnet.net, spa.gov.sa, al-marsd.com and tunisiatv.tn. youm7.com gives GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended their own groups with “Allow: /”. spa.gov.sa went the other way round: it blocked GPTBot and CCBot in early 2026 and now names both with “Allow: /”.

Some sites state a reservation without blocking. 6 sites carry a “Content-Signal” line with ai-train=no, the format Cloudflare published for saying how content may be used. 4 of them also block crawlers. al-marsd.com and royanews.tv do not. al-marsd.com lets a list of AI crawlers in under a comment that reads “allowed to read, not allowed to train”. royanews.tv allows GPTBot by name in the same file that says ai-train=no.

llms.txt, a plain-text guide written for language models, is still rare. 9 of the 91 sites we could check (9.9%) publish one: akhbaar24.com, annahar.com, asharq.com, babnet.net, elnashra.com, lapresse.tn, palinfo.com, sabq.org and youm7.com. sabq.org is the only site that both blocks GPTBot and publishes an llms.txt.

How has blocking changed since August 2023?

It started late and stayed low. In December 2023, 3 of 95 sites with an archived file (3.2%) blocked GPTBot, and none of them named it. The first site in the sample to block GPTBot by name was aljazeera.com, in June 2024. The share peaked at 15 of 98 (15.3%) in March 2026 and stood at 11 of 98 on 21 October 2026.

The three blockers of late 2023, addustour.com, alayam.com and aps.dz, all had default-deny files with Google’s crawlers allowed by name. GPTBot was caught by the wildcard, not singled out.

Named blocks came later: of the 20 sites that blocked GPTBot by name at some point, 2 did so in 2024 and 18 from June 2025 onwards. The count jumped in the second half of 2025, from 7 sites in August to 10 in September and 12 in October.

Why do blocks appear and disappear?

Many blocks did not last. 31 of the 101 sites with archived history blocked at least one AI crawler at some point between August 2023 and October 2026. 18 of those later dropped every AI block. Read any snapshot of this sector, ours included, with its date attached.

Three kinds of file account for the 18 drops.

• 10 were Cloudflare’s managed robots.txt. 17 of the 101 sites served it at some point. It is easy to spot: a legal preamble that begins “As a condition of accessing this website” and a group for CloudflareBrowserRenderingCrawler. In the versions we saw it blocks 8 of our 13 crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent and Amazonbot. Cloudflare offers it as a setting, and in the archive it tends to appear in one month and be gone in a later one. On 21 October 2026, 3 of the 13 blocking sites served it.
• 5 were blanket default-deny files that shut out every crawler not on an allow-list. aps.dz had one from its first archived file in September 2023 until September 2025, and annahar.com from December 2024 to November 2025.
• 3 were rules the site had written by name, at khaberni.com, medias24.com and spa.gov.sa.

Four sites served a file that shut out every crawler, Googlebot included, in at least one monthly capture: two in December 2024, one in June 2025 and one in the first capture of October 2026. Each was gone at the next reading, which for the October 2026 case was our census on 21 October.

A robots.txt file is the publisher’s own decision, and a short rule is as legitimate as a permanent one. We report these four cases because they move the monthly counts, and because anyone checking a site’s policy should know a single capture can catch a file like this.

The archive and our census agree on 18 of the 20 sites that have both an October 2026 capture and a census reading. One of the two that differ is almamlakatv.com, with no AI rules in the archive’s October capture and Cloudflare’s managed block on 21 October. The other is the October 2026 case described above.

How do Arab news sites compare with the Reuters Institute study?

Far fewer block. Reuters counts a site as blocking OpenAI if it disallows GPTBot, ChatGPT-User or all crawlers. On that definition, 3 of the 95 Arab sites with an archived file for December 2023 (3.2%) blocked OpenAI and 3 (3.2%) blocked Google-Extended. Reuters found 48% and 24% across ten countries at the end of 2023.

Reuters’ country figures for OpenAI ran from 79% in the United States to 20% in Mexico and Poland. For Google they ran from 60% in Germany to 7% in Poland and Spain. Even the lowest OpenAI figure in that set is more than six times the Arab one.

Reuters also found that no site unblocked during 2023. That holds in our archive for 2023 too, and the 18 drops all came later.

The comparison is context, not a matched test. Reuters took the 15 most used online news brands in each country from its Digital News Report survey and the most common robots.txt in each month. We took publishers from Semrush traffic lists and the first archived file in each month. Both studies use the Wayback Machine and the same blocking definition, which is why we report the December 2023 figure this way.

Country by country

Bahrain, Jordan and Morocco had the most GPTBot blockers, 4 each. In Egypt and Tunisia the only GPTBot blocker in the sample was Al Jazeera’s Arabic site. Each country has 10 to 15 readable sites, so a single site moves its share by up to 10 points, and we show counts rather than country percentages.

Counts of readable sites on 21 October 2026. A site that ranks in several countries is counted in each. Oman’s list had 11 qualifying news publishers and the UAE’s 13; the others had 15. Unreadable sites are listed in the method.

What this means for publishers

Blocking AI crawlers is a legitimate choice, and most of the region’s leading sites have not made it in either direction. If you do decide, decide crawler by crawler. Training crawlers and answer crawlers do different jobs, and a CDN default can block some of them for you without anyone in the newsroom writing a rule.

• Read your live file, not the one in your CMS. 10 of the 18 drops in our archive were Cloudflare’s managed block appearing and disappearing. Fetch yourdomain.com/robots.txt from outside your network and check which crawlers it names.
• Separate training from answers. Blocking GPTBot does not block OAI-SearchBot or ChatGPT-User, and OpenAI says a site that disallows OAI-SearchBot is not shown in ChatGPT search answers. Google says Google-Extended does not affect a site’s inclusion or ranking in Google Search. Decide each one on purpose.
• Check the allow-list in a default-deny file. 4 sites in our archive served, at least once, a file that also shut out Googlebot.
• Remember that robots.txt is a request. 16 of the 107 domains refused or did not answer a plain script asking for robots.txt itself, which shows how much happens at the firewall. If enforcement matters to you, robots.txt is the statement of intent and the firewall is the lock.
• Write the date down. Log when you change the file and why, so that you can later tell whether traffic or AI referrals moved after it.

What this means for brands that want to be cited in Arabic AI answers

For brands, the open door is the finding. 81 of 98 leading Arab news sites have no rule for any AI crawler, and only 4 block OAI-SearchBot. For crawlers that respect robots.txt, coverage on most of these outlets is readable. We did not test which outlets AI assistants cite in Arabic. That is a separate question, and this study only shows who has closed the door.

• Earned coverage on most top Arab outlets is not walled off by robots.txt. Placements there can be read by AI crawlers that follow the file, which is the first condition for being used in an answer.
• Check an outlet’s file before a placement aimed at AI answers. 6 of the 98 sites block at least one of OAI-SearchBot, ChatGPT-User and PerplexityBot. Al Jazeera’s two sites, for example, allow OAI-SearchBot but block ChatGPT-User and PerplexityBot.
• Recheck, because files change. 18 sites dropped their AI blocks within our window, and a few added them in the last months.
• Do not treat llms.txt as a requirement. 9 of 91 top news sites publish one. It is not what decides whether a page can be read.

Our read, from the lead researcher

“The numbers say most Arab newsrooms have not yet treated robots.txt as an editorial decision. 10 of the 18 dropped blocks trace back to one CDN setting, and 4 sites shut out Google itself in at least one monthly capture. That is a file being run as infrastructure. The 3 sites that block training and keep answer crawlers open are the ones that have clearly decided something, written it down and kept it. Whatever a newsroom chooses, that is the habit I would copy.” Beshoy Adel, Head of SEO and GEO, VOCTOS

What this does not prove

• robots.txt shows intent, not enforcement. Crawlers choose whether to obey it, and a firewall or CDN can block a crawler that robots.txt allows. We did not measure firewall rules. 16 domains refused or ignored our own script, so edge blocking is common in this sample.
• The archive has gaps. 6 of the 107 domains have no usable Wayback history, and 62 of the 101 with history have no capture after May 2026. We leave June to September 2026 out of the charts for that reason. A month with no capture takes its neighbours’ state only when the captures on either side agree.
• One capture per month. We used the first archived file in each month, not the one nearest the 15th as pre-registered, so a change late in a month shows up a month later.
• The sample comes from Semrush traffic estimates, which measure search-driven visits, not total readership. A site in several countries’ lists is counted once in the totals and once in each country.
• A missing robots.txt counts as allow-all, a decision logged during the census. It matches how crawlers read a 404.
• We cannot see motives. From outside, a deliberate editorial decision, a CDN default and a deployment mistake look the same. We describe files, not intentions.
• We did not measure outcomes. This study does not show whether AI companies honour these files, whether blocked content appears in AI answers, or whether any site has a licensing deal.

Method

Sample

We took Semrush’s public “Top Websites” list for the Newspapers category in each Arab country with a Semrush database, September 2026 edition, as shown on 21 October 2026. Libya has a database but no published list, which leaves 12 countries. From the first 100 domains in each list we kept the first 15 Arab news publishers, in rank order. A publisher qualified if it covers general or business news and current affairs and is Arabic-language or owned or based in an Arab country.

Platforms, aggregators without their own reporting, sports, film, religious, weather and price sites were excluded, and the exclusions are listed with the data. That gives 174 country and site pairs and 107 distinct domains.

Today’s census

On 21 October 2026 a script requested robots.txt and llms.txt from every domain, www first, with a desktop Chrome user agent and a pause between sites.

16 domains returned HTTP 403 or did not answer. We opened them in a normal browser: 7 could be read that way, and 9 stayed unreadable and are left out of today’s shares: alaraby.co.uk, khaberni.com, kuna.net.kw, lebanonfiles.com, lematin.ma, medias24.com, omannews.gov.om, raya.com and sba.sa. The 11 llms.txt hits were re-read once on 6 November 2026, which removed two HTML pages the census had accepted.

Parsing

Every saved file was read with a parser written to Google’s robots.txt specification. User-agent names are matched exactly, ignoring case, and every group naming a crawler is merged. A crawler with no group of its own follows the “*” groups. The longest matching rule wins, Allow wins a tie, and “*” and “$” in paths are honoured.

A crawler counts as blocked when “/” is disallowed for it, which covers “Disallow: /” and “Disallow: /*”. A group that names a crawler and disallows only some paths is recorded as partial (one site). The first census used Python’s standard parser, which matches names by substring. Switching changed two verdicts, both logged.

History

For each domain we asked the Wayback Machine’s CDX index for robots.txt captures with HTTP status 200, August 2023 to October 2026. We kept one per month, the first in the month, from the www host or, if it had none, the bare domain. That produced 3,007 usable monthly files for 101 domains.

Empty files (154) and bot-challenge pages (7) count as missing. HTML pages served at /robots.txt (28) count as no robots.txt. The census of 21 October closes the series for the 73 domains with no October 2026 capture.

Deviations and code

18 deviations from the pre-registration are logged, each with its effect on the numbers. The largest are the first-capture rule, the gap rule, the parser change and the llms.txt re-read. The analysis script is deterministic and every number on this page is checked against its output by a second script before publication.

Questions people ask about this

How many Arab news websites block ChatGPT’s crawlers?

On 21 October 2026, 11 of 98 leading Arab news sites (11.2%) blocked GPTBot, OpenAI’s training crawler. Fewer blocked the crawlers behind ChatGPT’s answers: 5 blocked ChatGPT-User and 4 blocked OAI-SearchBot. The sample is the top news publishers on Semrush’s lists for 12 Arab countries.

Do Arab news sites block AI crawlers more or less than Western publishers?

Less. At the end of 2023, 3 of 95 Arab sites with an archived robots.txt (3.2%) blocked OpenAI under the Reuters Institute’s definition. Reuters found 48% of the most used news sites in ten countries, including the US, UK and Germany, blocking at the same point. The samples were built differently, so treat it as context.

Which AI crawler do Arab news sites block most?

Bytespider, ByteDance’s crawler, blocked by 12 of 98 sites (12.2%) on 21 October 2026. GPTBot and ClaudeBot followed with 11 each. OAI-SearchBot, which OpenAI uses for ChatGPT search, was blocked least, by 4 sites (4.1%).

Does blocking GPTBot keep a news site out of ChatGPT answers?

Not on its own. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, and says that sites which disallow OAI-SearchBot are not shown in ChatGPT search answers. In our sample 11 sites block GPTBot but only 4 block OAI-SearchBot. We did not test what ChatGPT then shows.

How many Arab news sites have an llms.txt file?

9 of the 91 sites we could check (9.9%) on 21 October 2026, among them youm7.com, asharq.com and sabq.org. Two other sites returned an HTML page at /llms.txt and were not counted.

Is robots.txt enough to stop AI crawlers?

No. robots.txt is a public request that crawlers choose to follow. Firewalls and CDNs can block crawlers whatever the file says, and 16 of the 107 domains in this study refused or ignored a plain script asking for robots.txt itself. This study measures what the files say, not what gets through.

Data, credits and how to cite

Download the dataset (CSV). One row per domain. The columns cover countries and Semrush ranks, how the file was read, whether a robots.txt exists, and the status of each of the 13 crawlers on 21 October 2026 (blocked, blocked by wildcard, partial, allowed or unknown). They also cover llms.txt, the Content-Signal reservation, archive coverage and the first month each crawler was seen blocked (“<=” marks a site already blocking in its first archived file).

The dataset is released under the CC BY 4.0 licence: reuse it freely, crediting VOCTOS with a link to this page.

The monthly series, the pre-registration and the deviation log are available on request.

Cite as: VOCTOS (2026). Arab news sites and AI crawlers: robots.txt rules at 107 leading news sites in 12 Arab countries, August 2023 to October 2026. Lead researcher: Beshoy Adel. https://www.voctos.com/research/arab-news-sites-ai-crawlers/

Credits: the design follows Richard Fletcher, “How many news websites block AI crawlers?”, Reuters Institute for the Study of Journalism, 22 February 2024, published under CC BY. The history comes from the Internet Archive’s Wayback Machine, and the sample from Semrush’s public Top Websites lists.

Corrections: if your site is in the sample and a reading here looks wrong, write to us with the date and the file you see. We will check it against the archive and publish any correction on this page.

If you want to know what AI assistants say when your customers ask about your category, our AI search audit runs those questions and shows you the answers.