{"id":7457,"date":"2026-08-07T09:05:00","date_gmt":"2026-08-07T09:05:00","guid":{"rendered":"https:\/\/www.voctos.com\/?p=7457"},"modified":"2026-10-01T21:40:09","modified_gmt":"2026-10-01T21:40:09","slug":"what-is-robots-txt-how-to-configure","status":"publish","type":"post","link":"https:\/\/www.voctos.com\/ar\/blog\/what-is-robots-txt-how-to-configure\/","title":{"rendered":"\u0645\u0627 \u0647\u0648 \u0645\u0644\u0641 robots.txt \u0648\u0643\u064a\u0641 \u062a\u064f\u0639\u062f\u0651\u0647 \u0628\u0646\u0641\u0633\u0643"},"content":{"rendered":"\n<div style=\"font:400 13px\/1.5 Arial,sans-serif;color:#98A2AC;margin:0 0 14px;\"><a href=\"https:\/\/www.voctos.com\/\" style=\"color:#98A2AC;text-decoration:none;\">VOCTOS<\/a> &rsaquo; <a href=\"https:\/\/www.voctos.com\/blog\/technical-seo-guide\/\" style=\"color:#4B4DE8;text-decoration:none;\">The Technical SEO Guide<\/a> &rsaquo; <span>This guide<\/span><\/div>\n\n\n<p><strong>robots.txt is a plain-text file placed in the root of your domain (for example, <code>https:\/\/example.com\/robots.txt<\/code>) that tells search engine crawlers such as Googlebot which URLs they may or may not request. You configure it yourself by creating a UTF-8 text file, adding a few directives (<code>User-agent<\/code>, <code>Disallow<\/code>, <code>Allow<\/code>, and <code>Sitemap<\/code>), uploading it to the root folder, and validating it in Google Search Console. It controls crawling, not indexing, so it is one piece of technical SEO, not the whole toolkit.<\/strong><\/p>\n\n<figure><img decoding=\"async\" src=\"https:\/\/www.voctos.com\/wp-content\/uploads\/2026\/08\/robots-txt-in-website-root-directing-crawlers.webp\" alt=\"Diagram of a robots.txt file in a website root directory directing search engine crawlers\"\/><\/figure>\n\n<h2>Key takeaways<\/h2>\n<ul>\n  <li>robots.txt is a public file in your domain root that gives crawling instructions to search engine bots, it must be reachable at <code>\/robots.txt<\/code> and encoded in UTF-8.<\/li>\n  <li>The four directives you will use most are <code>User-agent<\/code>, <code>Disallow<\/code>, <code>Allow<\/code>, and <code>Sitemap<\/code>. Wildcards <code>*<\/code> and <code>$<\/code> let you match URL patterns.<\/li>\n  <li>robots.txt controls <em>crawling<\/em>, not <em>indexing<\/em>. A blocked page can still appear in Google&#8217;s results if other sites link to it. Use a <code>noindex<\/code> meta tag or HTTP header to keep a page out of the index.<\/li>\n  <li>Never block CSS and JavaScript that Google needs to render your pages, and never ship an accidental <code>Disallow: \/<\/code>it can wipe out your organic traffic.<\/li>\n  <li>Always test changes with the robots.txt report and URL Inspection tool in Google Search Console before and after you publish.<\/li>\n<\/ul>\n\n<h2>What robots.txt is and why it matters<\/h2>\n<p>robots.txt is a small text file that acts as a set of ground rules between your website and automated crawlers. The standard was first introduced in 1994 as the Robots Exclusion Protocol and was formally standardized in 2022 as RFC 9309, which Google helped author. Despite being only a few kilobytes, it is one of the first things a well-behaved crawler requests when it visits your site.<\/p>\n<p>When configured correctly, robots.txt does three useful jobs:<\/p>\n<ul>\n  <li><strong>Manages crawl budget.<\/strong> It signals which low-value URLs, faceted filters, internal search results, session parameters, crawlers should skip, so Googlebot spends its limited time on pages that matter.<\/li>\n  <li><strong>Keeps clutter out of the crawl path.<\/strong> It steers bots away from admin panels, staging areas, and system folders.<\/li>\n  <li><strong>Points crawlers to your sitemap.<\/strong> The <code>Sitemap<\/code> directive helps search engines discover your important URLs faster.<\/li>\n<\/ul>\n<p>The stakes are real. A single stray rule can hide an entire site from search. In one widely cited incident, a large publisher accidentally disallowed all of its articles and watched organic traffic collapse almost overnight. A correctly written file is quietly foundational to every other part of your <a href=\"https:\/\/www.voctos.com\/seo\/\">SEO strategy<\/a>.<\/p>\n\n<h3>How crawlers read the file, and its limits<\/h3>\n<p>robots.txt lives at the root of the domain and must be served as a text file, ideally under 500 KiB, which is where Google stops reading. You can confirm it works by typing the URL into a browser, if you see the contents, the file is reachable.<\/p>\n<p>Just as important is understanding what robots.txt <em>cannot<\/em> do:<\/p>\n<ul>\n  <li><strong>It does not guarantee a page stays out of Google.<\/strong> Disallowed URLs can still be indexed if they are linked from elsewhere; Google may show them without a description.<\/li>\n  <li><strong>It does not protect private content.<\/strong> The file is public and readable by anyone, including competitors. Use authentication for anything truly sensitive.<\/li>\n  <li><strong>It does not influence rankings directly.<\/strong> Allowing a page to be crawled is not the same as making it rank.<\/li>\n<\/ul>\n\n<h2>Syntax and the core directives<\/h2>\n<p>robots.txt follows strict, line-by-line rules. One misplaced character can cause a crawler to ignore an instruction, so precision matters. Here are the directives you will actually use.<\/p>\n\n<h3>User-agent: which crawler the rules apply to<\/h3>\n<p>This directive names the bot a block of rules is written for. Use <code>*<\/code> to address all crawlers, or a specific token to target one.<\/p>\n<pre><code>User-agent: *          # applies to all crawlers\nUser-agent: Googlebot  # Google's main crawler only\nUser-agent: Bingbot    # Microsoft Bing's crawler only<\/code><\/pre>\n<p>If you omit <code>User-agent<\/code>, the block is invalid and will be ignored. You can write separate blocks for different bots.<\/p>\n\n<h3>Disallow: block crawling of a path<\/h3>\n<p>This tells the named crawler not to request URLs that begin with the given path. The leading slash is required.<\/p>\n<pre><code>Disallow: \/admin\/     # block the admin area\nDisallow: \/tmp\/       # block temporary files\nDisallow: \/*?sort=    # block sort-parameter URLs<\/code><\/pre>\n<p>An empty value, <code>Disallow:<\/code> with nothing after it, means &#8220;allow everything.&#8221;<\/p>\n\n<h3>Allow: carve out an exception<\/h3>\n<p>The <code>Allow<\/code> directive re-opens a specific path inside a broader <code>Disallow<\/code>. It is handy when you want to block a section but keep one URL crawlable.<\/p>\n<pre><code>User-agent: *\nDisallow: \/blog\/            # block the whole blog folder\nAllow: \/blog\/seo-tips\/      # but allow this one page<\/code><\/pre>\n<p>Google resolves conflicts by the most specific (longest) matching rule, so a targeted <code>Allow<\/code> can override a broader <code>Disallow<\/code>.<\/p>\n\n<h3>Sitemap: point to your XML sitemap<\/h3>\n<p>List one or more full, absolute sitemap URLs. This directive is independent of <code>User-agent<\/code> and helps crawlers find your content.<\/p>\n<pre><code>Sitemap: https:\/\/example.com\/sitemap.xml\nSitemap: https:\/\/example.com\/sitemap-images.xml<\/code><\/pre>\n\n<h3>Wildcards and comments<\/h3>\n<p>Two special characters give you pattern matching: <code>*<\/code> stands for any sequence of characters, and <code>$<\/code> anchors the end of a URL. The hash symbol <code>#<\/code> starts a comment.<\/p>\n<pre><code># block every PDF file\nDisallow: \/*.pdf$\n\n# block all internal search result pages\nDisallow: \/search?*<\/code><\/pre>\n\n<h2>Ready-made examples for WordPress and other CMS platforms<\/h2>\n<p>The right configuration depends on your platform. Below are practical starting templates. Adapt the paths to your real structure, and always test before relying on them.<\/p>\n\n<h3>WordPress<\/h3>\n<p>A typical WordPress site benefits from blocking admin and system endpoints while keeping media crawlable. If you run WordPress, our <a href=\"https:\/\/www.voctos.com\/web-development\/wordpress\/\">WordPress development team<\/a> can wire this up alongside your SEO plugin.<\/p>\n<pre><code>User-agent: *\nDisallow: \/wp-admin\/\nAllow: \/wp-admin\/admin-ajax.php\nDisallow: \/wp-includes\/\nDisallow: \/wp-content\/plugins\/\nDisallow: \/?s=                 # internal search results\nDisallow: \/comments\/feed\/\nAllow: \/wp-content\/uploads\/    # keep images and files crawlable\n\nSitemap: https:\/\/example.com\/wp-sitemap.xml<\/code><\/pre>\n<p>Note that <code>\/wp-admin\/admin-ajax.php<\/code> is explicitly allowed because themes and plugins rely on it, and <code>\/wp-content\/uploads\/<\/code> is kept open so images can be crawled. If you use Yoast or another SEO plugin, it generates a sitemap index (often <code>sitemap_index.xml<\/code>), reference that instead.<\/p>\n\n<h3>E-commerce store (Shopify, WooCommerce, and similar)<\/h3>\n<p>Online stores generate huge numbers of near-duplicate URLs through cart actions, accounts, and filters. Trim those from the crawl path.<\/p>\n<pre><code>User-agent: *\nDisallow: \/cart\nDisallow: \/checkout\nDisallow: \/account\nDisallow: \/*?sort=\nDisallow: \/*?filter=\nAllow: \/products\/\n\nSitemap: https:\/\/shop.example.com\/sitemap.xml<\/code><\/pre>\n\n<h3>Multilingual site<\/h3>\n<p>Keep language-specific admin areas closed and confirm each language has its own sitemap.<\/p>\n<pre><code>User-agent: *\nDisallow: \/en\/admin\/\nDisallow: \/de\/admin\/\n\nSitemap: https:\/\/example.com\/sitemap-en.xml\nSitemap: https:\/\/example.com\/sitemap-de.xml<\/code><\/pre>\n<p>Large marketplaces and SaaS platforms often generate robots.txt dynamically so that rules stay in sync with a changing URL structure: a task worth handing to your <a href=\"https:\/\/www.voctos.com\/technical-support\/\">technical support team<\/a>.<\/p>\n\n<figure><img decoding=\"async\" src=\"https:\/\/www.voctos.com\/wp-content\/uploads\/2026\/08\/search-console-robots-txt-report-testing-directives.webp\" alt=\"Google Search Console robots.txt report showing directives being tested against a URL\"\/><\/figure>\n\n<h2>Common mistakes and how to fix them<\/h2>\n<p>Even experienced developers slip up in robots.txt, and the cost can be steep. Watch for these.<\/p>\n<ul>\n  <li><strong>An accidental full block.<\/strong> <code>Disallow: \/<\/code> under <code>User-agent: *<\/code> tells every crawler to skip your entire site. This is the classic cause of a sudden traffic collapse after a launch or migration.<\/li>\n  <li><strong>Blocking CSS and JavaScript.<\/strong> If Googlebot cannot fetch the assets it needs to render a page, it may misjudge the layout and mobile-friendliness. Keep rendering resources crawlable.<\/li>\n  <li><strong>Missing the leading slash.<\/strong> <code>Disallow: admin<\/code> is not a valid path; write <code>Disallow: \/admin\/<\/code>.<\/li>\n  <li><strong>Relying on robots.txt to remove a page from Google.<\/strong> Because a blocked page can still be indexed via inbound links, use <code>noindex<\/code> instead (see below).<\/li>\n  <li><strong>Case and whitespace errors.<\/strong> The filename must be lowercase <code>robots.txt<\/code>, and stray spaces after a colon can break parsing for stricter crawlers.<\/li>\n<\/ul>\n<p>Whenever you inherit or overhaul a site, a full <a href=\"https:\/\/www.voctos.com\/seo\/audit\/\">SEO audit<\/a> should include a review of the robots.txt file so these issues are caught before they cost you rankings.<\/p>\n\n<h2>How robots.txt differs from noindex and the meta robots tag<\/h2>\n<p>This is the single most misunderstood point in technical SEO, so it is worth being precise.<\/p>\n<ul>\n  <li><strong>robots.txt controls crawling.<\/strong> It decides whether a bot is allowed to <em>request<\/em> a URL. It works at the site or directory level.<\/li>\n  <li><strong>The <code>noindex<\/code> meta robots tag (or <code>X-Robots-Tag<\/code> HTTP header) controls indexing.<\/strong> It tells Google not to <em>keep<\/em> a page in its index. It works at the individual page level.<\/li>\n<\/ul>\n<p>The critical interaction: if you block a page in robots.txt, Google never crawls it, which means Google never sees the <code>noindex<\/code> tag on that page. So a page that is both disallowed and marked <code>noindex<\/code> can paradoxically stay in the index. To reliably remove a URL, <em>allow<\/em> crawling and add <code>noindex<\/code>, or use the URL removal tool in Search Console. A simple page-level tag looks like this:<\/p>\n<pre><code>&lt;meta name=\"robots\" content=\"noindex, follow\"&gt;<\/code><\/pre>\n<p>For non-HTML files such as PDFs, use the header equivalent:<\/p>\n<pre><code>X-Robots-Tag: noindex<\/code><\/pre>\n<p>In short: use robots.txt to save crawl budget on low-value URLs, and use <code>noindex<\/code> to keep specific pages out of search results. They solve different problems and often work together.<\/p>\n\n<h2>How to test robots.txt in Google Search Console<\/h2>\n<p>Never publish a robots.txt change blind. Google Search Console gives you the tools to see the file the way Googlebot does.<\/p>\n<ol>\n  <li><strong>Open the robots.txt report.<\/strong> In Search Console, go to <em>Settings &rarr; robots.txt<\/em>. Google shows the version it last fetched, when it was fetched, and flags any parsing errors or warnings.<\/li>\n  <li><strong>Request a re-fetch after changes.<\/strong> If you have just uploaded a new file, use the report&#8217;s option to ask Google to fetch the latest version so it stops serving a cached copy.<\/li>\n  <li><strong>Test specific URLs with URL Inspection.<\/strong> Paste a URL into the inspection bar at the top of Search Console. It tells you whether the page is allowed or blocked by robots.txt, and whether it is indexed.<\/li>\n  <li><strong>Recheck after edits.<\/strong> Confirm that pages you want indexed are crawlable and that the URLs you intended to block are actually blocked, and nothing else was caught by accident.<\/li>\n<\/ol>\n<p>Make this a habit: run the check after every redesign, CMS migration, or structural change, and audit the file at least once a quarter. As search shifts toward AI-driven answer engines, the same discipline of clean, crawlable technical foundations underpins <a href=\"https:\/\/www.voctos.com\/geo\/\">generative engine optimization (AI SEO)<\/a> too.<\/p>\n\n<h2>Frequently asked questions<\/h2>\n\n<h3>Do I need a robots.txt file at all?<\/h3>\n<p>Not strictly. If you have no robots.txt file, crawlers assume they can access everything, which is fine for many small sites. However, having one lets you manage crawl budget, point to your sitemap, and keep bots out of system folders, so most sites benefit from a deliberate, minimal file rather than none.<\/p>\n\n<h3>Will Disallow remove a page from Google search results?<\/h3>\n<p>No. <code>Disallow<\/code> stops Google from crawling a page, but the URL can still appear in results if other pages link to it. To reliably remove a page, allow crawling and add a <code>noindex<\/code> meta tag or <code>X-Robots-Tag<\/code> header, or use the removal tool in Google Search Console.<\/p>\n\n<h3>Where exactly does the robots.txt file go?<\/h3>\n<p>It must sit in the root of your domain and be reachable at <code>https:\/\/yourdomain.com\/robots.txt<\/code>. A file placed in a subfolder will not be recognized. Each subdomain needs its own robots.txt, and the filename must be all lowercase.<\/p>\n\n<h3>Can I set different rules for Googlebot and other crawlers?<\/h3>\n<p>Yes. Write a separate <code>User-agent<\/code> block for each crawler you want to treat differently, for example, one block for <code>Googlebot<\/code> and another for <code>Bingbot<\/code>. A crawler follows the most specific block that names it, and falls back to the <code>User-agent: *<\/code> block only if no block names it directly.<\/p>\n\n<h3>How often should I review my robots.txt?<\/h3>\n<p>Audit it at least once a quarter, and always after a redesign, a platform migration, or any change to your URL structure. Check the robots.txt report in Search Console and your server logs to confirm crawlers are spending time on the right pages.<\/p>\n\n<script type=\"application\/ld+json\">\n{\n  \"@context\": \"https:\/\/schema.org\",\n  \"@type\": \"FAQPage\",\n  \"mainEntity\": [\n    {\n      \"@type\": \"Question\",\n      \"name\": \"Do I need a robots.txt file at all?\",\n      \"acceptedAnswer\": {\n        \"@type\": \"Answer\",\n        \"text\": \"Not strictly. If you have no robots.txt file, crawlers assume they can access everything, which is fine for many small sites. However, having one lets you manage crawl budget, point to your sitemap, and keep bots out of system folders, so most sites benefit from a deliberate, minimal file rather than none.\"\n      }\n    },\n    {\n      \"@type\": \"Question\",\n      \"name\": \"Will Disallow remove a page from Google search results?\",\n      \"acceptedAnswer\": {\n        \"@type\": \"Answer\",\n        \"text\": \"No. Disallow stops Google from crawling a page, but the URL can still appear in results if other pages link to it. To reliably remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header, or use the removal tool in Google Search Console.\"\n      }\n    },\n    {\n      \"@type\": \"Question\",\n      \"name\": \"Where exactly does the robots.txt file go?\",\n      \"acceptedAnswer\": {\n        \"@type\": \"Answer\",\n        \"text\": \"It must sit in the root of your domain and be reachable at https:\/\/yourdomain.com\/robots.txt. A file placed in a subfolder will not be recognized. Each subdomain needs its own robots.txt, and the filename must be all lowercase.\"\n      }\n    },\n    {\n      \"@type\": \"Question\",\n      \"name\": \"Can I set different rules for Googlebot and other crawlers?\",\n      \"acceptedAnswer\": {\n        \"@type\": \"Answer\",\n        \"text\": \"Yes. Write a separate User-agent block for each crawler you want to treat differently, for example one block for Googlebot and another for Bingbot. A crawler follows the most specific block that names it, and falls back to the User-agent: * block only if no block names it directly.\"\n      }\n    },\n    {\n      \"@type\": \"Question\",\n      \"name\": \"How often should I review my robots.txt?\",\n      \"acceptedAnswer\": {\n        \"@type\": \"Answer\",\n        \"text\": \"Audit it at least once a quarter, and always after a redesign, a platform migration, or any change to your URL structure. Check the robots.txt report in Search Console and your server logs to confirm crawlers are spending time on the right pages.\"\n      }\n    }\n  ]\n}\n<\/script>\n\n<!--voctos-pillar-TheTechnicalSEOGuide-->\n\n<div style=\"border-left:4px solid #4B4DE8;background:#F8FAFC;padding:15px 19px;margin:26px 0;border-radius:0 10px 10px 0;\"><div style=\"font:700 12px\/1 Arial,sans-serif;color:#4B4DE8;letter-spacing:.1em;margin-bottom:7px;\">PART OF THE TECHNICAL SEO GUIDE<\/div><div style=\"font:400 16px\/1.7 Arial,sans-serif;color:#3C464F;\">This guide is one of 16 in <a href=\"https:\/\/www.voctos.com\/blog\/technical-seo-guide\/\" style=\"color:#4B4DE8;font-weight:700;\">The Technical SEO Guide<\/a>, our full library on the topic.<\/div><\/div>\n\n\n<!--voctos-related-->\n<h2>Related guides<\/h2>\n<ul><li><a href=\"https:\/\/www.voctos.com\/blog\/broken-links-find-and-fix\/\">Broken Links: How to Find and Fix Them Yourself<\/a><\/li><li><a href=\"https:\/\/www.voctos.com\/blog\/robots-txt-ai-crawlers\/\">robots.txt for AI Crawlers: Which AI Bots to Allow or Block (2026)<\/a><\/li><li><a href=\"https:\/\/www.voctos.com\/blog\/website-navigation\/\">Website Navigation: What It Is and How to Use It<\/a><\/li><li><a href=\"https:\/\/www.voctos.com\/blog\/what-are-duplicate-pages\/\">What Are Duplicate Pages and Why Are They Dangerous<\/a><\/li><\/ul>","protected":false},"excerpt":{"rendered":"<p>Learn what robots.txt is and how to configure it yourself: core directives, WordPress and CMS examples, common mistakes, and how to test it in Search Console.<\/p>","protected":false},"author":1,"featured_media":13776,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[43],"tags":[],"class_list":["post-7457","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-seo"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/posts\/7457","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/comments?post=7457"}],"version-history":[{"count":1,"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/posts\/7457\/revisions"}],"predecessor-version":[{"id":13322,"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/posts\/7457\/revisions\/13322"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/media\/13776"}],"wp:attachment":[{"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/media?parent=7457"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/categories?post=7457"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.voctos.com\/ar\/wp-json\/wp\/v2\/tags?post=7457"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}