To create robots txt is to build a plain-text configuration file — named robots.txt — and place it at the root of your website so that search engine crawlers know exactly which pages to visit and which to skip. It is one of the most fundamental, yet frequently misunderstood, tools in technical SEO.
Every time Googlebot or another crawler arrives at your domain, the very first file it looks for is robots.txt. Get it right, and you guide crawlers efficiently through your most valuable content. Get it wrong, and you risk blocking entire sections of your site — or wasting precious crawl budget on pages that add no SEO value. This guide walks you through everything you need to know, from the basics of syntax to advanced strategies that professionals use daily.
What Is a Robots TXT File and Why Does It Matter?
A robots.txt file is a plain-text document that follows the Robots Exclusion Standard, a protocol established in 1994 that defines how web crawlers should interact with website content. It lives at the root of your domain — always accessible at https://yourdomain.com/robots.txt — and is read before any other page is crawled.
The file matters for three core reasons: crawl budget management, content privacy, and duplicate content prevention. Large websites with thousands of pages especially benefit from tight robots.txt control, ensuring crawlers spend time on high-value URLs rather than admin panels, login pages, or parameter-heavy filtered URLs.
A properly structured robots.txt file uses clean, readable directives that search engines process before crawling any page.
How to Create Robots TXT: Step-by-Step
Creating a robots.txt file requires nothing more than a plain-text editor. Follow these steps carefully to build one that is both valid and strategically effective.
Open a plain-text editor
Use Notepad (Windows), TextEdit in plain-text mode (Mac), or any code editor like VS Code. Never use a word processor — it will add hidden formatting characters that break the file.
Declare your User-agent
Every block begins with a User-agent: line. Use * to target all crawlers, or specify individual bots such as Googlebot or Bingbot.
Add Disallow and Allow rules
Use Disallow: /path/ to block a directory and Allow: /path/page to carve out exceptions within a blocked section. A blank Disallow: value means all content is permitted.
Add your Sitemap reference
At the bottom of the file, include a Sitemap: directive pointing to your XML sitemap. This is not mandatory, but it is strongly recommended as it helps crawlers discover your full content structure.
Save as robots.txt and upload
Save the file with the exact name robots.txt (all lowercase). Upload it via FTP, SFTP, or your CMS file manager to the root directory of your site — not a subfolder.
Robots TXT Syntax: A Real-World Example
Below is a practical example suitable for most websites. It allows all crawlers full access while blocking common low-value directories and declaring the sitemap location.
# All crawlers — global rules User-agent: * Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /search? Allow: /wp-admin/admin-ajax.php # Googlebot — specific rules User-agent: Googlebot Disallow: /staging/ Allow: / # Sitemap location Sitemap: https://yourdomain.com/sitemap.xml
Understanding the Key Directives
Planning your site structure before you create robots txt rules helps ensure no valuable pages are accidentally blocked.
Common Robots TXT Mistakes to Avoid
Even experienced developers make critical errors when managing robots.txt. Here are the most damaging mistakes and how to sidestep them.
❌ Blocking your entire site
A single misplaced Disallow: / under User-agent: * will block all crawlers from every page. This is surprisingly common after CMS migrations or staging-to-live deployments.
❌ Treating robots.txt as a security tool
Robots.txt is a public file. Anyone can read it. Never list sensitive file paths or admin URLs hoping to hide them — you are effectively advertising them to bad actors.
❌ Blocking CSS and JavaScript files
Google needs to render your pages to understand them. Blocking /wp-content/ or key JS/CSS directories prevents proper rendering and can lower your rankings significantly.
❌ Assuming robots.txt removes pages from search results
Blocking a URL only prevents crawling — it does not remove an existing indexed URL. For guaranteed de-indexing, use a noindex meta tag or Google Search Console’s URL removal tool.
Robots TXT for WordPress Sites
WordPress generates a virtual robots.txt file automatically if no physical one exists. However, relying on the default is rarely optimal. Plugins like Yoast SEO or Rank Math allow you to edit the robots.txt file directly from your dashboard under Tools > File Editor, giving you full control without FTP access.
For WordPress, a well-optimised robots.txt should typically block /wp-admin/ (except admin-ajax.php), tag archives, author pages (to prevent thin content penalties), and URL parameters generated by plugins or analytics tools.
How to Test Your Robots TXT File
After you create robots txt rules, always test them before relying on them. Google Search Console includes a built-in robots.txt tester (found under the legacy tools section) that shows you exactly which URLs are blocked or allowed according to your current file. You can also use third-party tools that simulate how different crawlers interpret your directives.
Simply navigate to https://yourdomain.com/robots.txt in your browser to confirm the file is live and readable. If you see a 404 error, the file is not in the correct location.
Robots TXT and Crawl Budget Optimisation
For large or growing websites, crawl budget — the number of pages Googlebot will crawl within a given timeframe — is a genuine ranking factor. By strategically blocking low-value URLs such as faceted navigation pages, internal search results, and duplicate parameter-based URLs, you free up crawl budget for content that actually drives traffic and conversions.
This principle connects directly to broader content strategy. Resources like the guide on the impact of content length on SEO rankings at Rank Authority illustrate how the depth and quality of your crawlable pages affects where you appear in search results — making it even more important that crawlers reach your best work.
Visualising crawl paths helps clarify which sections benefit most from robots.txt filtering.
Frequently Asked Questions
What does it mean to create robots txt?
To create robots txt means to build a plain-text file named robots.txt placed at the root of your website that instructs search engine crawlers which pages or sections they are allowed or forbidden to access and index. It is the first file any compliant crawler reads upon visiting your domain.
Where should I place my robots.txt file?
Your robots.txt file must be placed at the root domain level, accessible at https://yourdomain.com/robots.txt. It cannot be placed in subdirectories and still apply site-wide.
Can robots.txt block pages from appearing in Google?
Blocking a page with robots.txt prevents crawlers from accessing its content, but it does not guarantee removal from search results. Google may still index a URL if other pages link to it. Use a noindex meta tag for guaranteed exclusion.
What is the difference between Disallow and Allow?
Disallow tells crawlers not to access a specific path or file, while Allow explicitly permits access to a path that might otherwise be blocked by a broader Disallow rule. Allow takes precedence when both rules match the same URL.
How often should I update my robots.txt file?
Review and update your robots.txt whenever your site structure changes, new sections are added, or you launch new content campaigns. Regular audits — much like the practice of updating content for SEO — ensure crawlers are always directed efficiently to your most valuable pages.
Final Thoughts: Why You Should Create Robots TXT With Care
To create robots txt correctly is to lay the foundation for every other SEO effort you make. It controls what crawlers see, how efficiently they explore your site, and ultimately which pages have the opportunity to rank. A well-crafted robots.txt file is invisible to users but profoundly influential in search. Audit it regularly, test every change before deploying, and treat it as a living document that evolves alongside your site. The few minutes it takes to get it right can protect years of SEO work.
