robots.txt Explained
Sometimes running a WordPress blog feels less like publishing and more like opening a small store on a hidden street.
You write useful posts, choose keywords carefully, add images, submit your sitemap, and wait for Google to notice. Then you open Google Search Console and see phrases like “Blocked by robots.txt,” “Discovered – currently not indexed,” or “Crawled – currently not indexed.” Suddenly, a tiny technical file you have never touched starts to look very important.
That file is called robots.txt.
At first glance, robots.txt looks like something only developers should care about. But for bloggers, affiliate site owners, niche publishers, and WordPress operators, it can quietly affect how search engines explore your site. Used correctly, it helps crawlers understand where to go. Used carelessly, it can block important pages, images, or resources from being crawled.
The tricky part is this: robots.txt does not work the way many beginners think it does. It does not “delete” pages from Google. It does not protect private content. It does not replace noindex tags. It is simply a set of crawling instructions for search engine bots.
Once you understand that difference, technical SEO becomes much less scary.
What Is robots.txt?
robots.txt is a plain text file placed at the root of a website.
For example, if your site is:
https://example.com
Then your robots.txt file is usually located at:
https://example.com/robots.txt
Search engine crawlers such as Googlebot and Bingbot check this file before crawling a site. The file tells them which parts of the site they are allowed to request and which parts they should avoid.
A basic WordPress robots.txt file often looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml
This setup tells all crawlers that the WordPress admin area should not be crawled, while still allowing the admin-ajax.php file, which some WordPress themes and plugins may need. It also shows search engines where the sitemap is located.
The most important thing to remember is this:
robots.txt controls crawling, not indexing.
That one sentence solves half of the confusion around robots.txt.
Why robots.txt Matters for WordPress SEO
For a small personal blog with ten posts, robots.txt may not seem like a big deal. But once a WordPress site grows into hundreds or thousands of posts, crawl management becomes more important.
Search engines do not crawl every site with unlimited attention. They spend crawl resources based on site quality, server response, internal links, update frequency, and technical signals. If your site creates many low-value URLs, such as internal search result pages, feed URLs, author archives, tag archives, or parameter URLs, search engines may spend time exploring pages that do not matter much.
That is where robots.txt can help.
It can reduce unnecessary crawling of areas like:
- WordPress admin pages
- Internal search result URLs
- Duplicate technical paths
- Staging or test sections
- Certain parameter-based URLs
However, this does not mean you should block everything that looks unimportant. Many site owners make the mistake of turning robots.txt into a giant wall. In SEO, blocking too much can create more problems than it solves.
A good robots.txt file is usually simple, intentional, and easy to understand.
Basic robots.txt Rules
Here are the most common robots.txt directives.
| Directive | Meaning | Example | Blogger’s Note |
|---|---|---|---|
User-agent | Specifies the crawler | User-agent: * | * means all crawlers |
Disallow | Blocks crawling of a path | Disallow: /wp-admin/ | Does not remove the URL from Google |
Allow | Allows a specific path | Allow: /wp-admin/admin-ajax.php | Useful for exceptions |
Sitemap | Shows sitemap location | Sitemap: https://example.com/sitemap_index.xml | Helpful for discovery |
Crawl-delay | Requests slower crawling | Crawl-delay: 10 | Google does not support this rule |
The rule most bloggers use is Disallow. But it should be used carefully.
For example:
User-agent: *
Disallow: /wp-admin/
This tells all crawlers not to crawl the WordPress admin area.
But this:
User-agent: *
Disallow: /
blocks crawling of the entire site.
That single slash can cause a major SEO disaster if it appears on a live blog.
robots.txt vs noindex: The Difference Bloggers Must Know
Many WordPress users confuse robots.txt with noindex. They are related to search visibility, but they do completely different jobs.
| Feature | robots.txt | noindex |
|---|---|---|
| Main purpose | Controls crawling | Controls indexing |
| Where it appears | /robots.txt file | HTML meta tag or HTTP header |
| Can Google crawl the page? | Usually no, if blocked | Yes |
| Can it remove a page from search results? | Not reliably | Yes, when processed correctly |
| Best use case | Admin pages, low-value crawl paths | Thin pages, duplicate pages, tag archives |
| Common mistake | Using it to hide pages from Google | Blocking the page before Google can see noindex |
Here is the key problem.
Suppose you want to remove tag pages from Google search. You might think this is the solution:
User-agent: *
Disallow: /tag/
But if Googlebot cannot crawl /tag/, it may not be able to see a noindex tag on those pages. In some cases, the URL may still appear in search results if Google discovers it through links.
For pages you want removed from search results, noindex is usually the better tool.
For pages you simply do not want crawlers to waste time on, robots.txt can be useful.
A simple way to remember it:
robots.txt says “don’t crawl this.”
noindex says “don’t show this in search results.”
A Safe robots.txt Setup for Most WordPress Blogs
For many WordPress blogs, a clean and safe robots.txt setup is enough.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml
For Kori Think or a similar blogging and SEO-focused WordPress site, this is usually a good starting point.
It blocks the admin area, keeps necessary WordPress functionality accessible, and points search engines to the sitemap. It does not block posts, categories, images, CSS, JavaScript, or uploaded media.
That matters because modern search engines do not only read plain text. They also need to understand layout, links, images, and page rendering. If you block important files inside /wp-content/, your site may become harder for search engines to understand.
For example, this is risky:
User-agent: *
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /category/
Disallow: /tag/
Blocking /wp-content/ can interfere with images, themes, scripts, and uploaded files. Blocking categories or tags may also weaken how search engines understand your site structure.
If you do not want tag pages indexed, use your SEO plugin’s noindex settings instead of blindly blocking them in robots.txt.
Real Example 1: The Entire Blog Stops Being Crawled
One of the most dangerous robots.txt mistakes is this:
User-agent: *
Disallow: /
This tells search engines not to crawl anything on the site.
This can happen when a WordPress site is built in private mode and later launched without changing the setting. In WordPress, there is a setting that discourages search engines from indexing the site. Some staging sites, maintenance plugins, or SEO plugins may also create restrictive robots.txt rules.
The symptoms may look like this:
| Symptom | Possible Cause | Where to Check |
|---|---|---|
| New posts do not appear on Google | Full-site crawl block | yourdomain.com/robots.txt |
| Search Console shows robots.txt warnings | Important URLs are blocked | Page indexing report |
| Sitemap is submitted but pages are not crawled | Crawlers cannot access URLs | URL Inspection Tool |
| Search result shows only URL, no snippet | Page may be blocked from crawling | Google search results |
The first thing to do is simple. Open your browser and type:
https://yourdomain.com/robots.txt
Then look for Disallow: /.
If your live blog has that rule, remove it unless you intentionally want the entire site blocked from crawling.
Real Example 2: Images Are Not Getting Search Traffic
Many bloggers focus only on posts, but images can also bring traffic through Google Images, Discover surfaces, and visual search behavior.
In WordPress, uploaded images are usually stored under:
/wp-content/uploads/
So this rule can be a problem:
User-agent: *
Disallow: /wp-content/uploads/
If your image folder is blocked, search engines may have trouble crawling your images. For a recipe blog, travel blog, product review site, tutorial site, or visual niche blog, that can reduce potential search exposure.
A better approach is usually this:
- Keep uploaded images crawlable.
- Use descriptive file names.
- Add natural alt text.
- Compress images properly.
- Use WebP when appropriate.
- Redirect or noindex attachment pages if needed.
Do not block your entire image folder unless you have a specific reason.
Real Example 3: Internal Search Pages Start Appearing on Google
WordPress internal search URLs often look like this:
https://example.com/?s=seo
https://example.com/search/robots
These pages can be useful for visitors, but they are usually not valuable as Google search results. Nobody wants to search Google and land on another search result page.
For large blogs, internal search pages can create many low-value URLs. In that case, you may consider blocking them:
User-agent: *
Disallow: /?s=
Disallow: /search/
But be careful. If those URLs are already indexed and you want them removed, noindex may be more appropriate first.
The practical rule is:
Use robots.txt to reduce crawling.
Use noindex to remove pages from search results.
Kori’s Mid-Article Thoughts
robots.txt can feel intimidating because it looks technical, but the real idea is simple.
The goal is not to build the most complicated file.
The goal is to avoid blocking the wrong things.
For a blog, the default attitude should be openness.
Let search engines reach your articles, images, categories, and important resources.
Then block only the paths that clearly do not help search visibility.
That mindset keeps the site clean without accidentally hiding valuable content.
One-Line Tip
After editing robots.txt, always open yourdomain.com/robots.txt in a browser and test important URLs in Google Search Console’s URL Inspection Tool.
WordPress robots.txt Checklist
Before changing robots.txt, review this checklist.
| Checkpoint | Question | Recommended Action |
|---|---|---|
| Full-site block | Is Disallow: / present? | Remove it on a live public blog |
| Admin area | Is /wp-admin/ blocked? | Usually fine |
| Ajax file | Is admin-ajax.php allowed? | Recommended for WordPress |
| Image uploads | Is /wp-content/uploads/ blocked? | Avoid blocking if image traffic matters |
| Sitemap | Is the sitemap URL correct? | Keep it updated |
| Tags and archives | Are they blocked in robots.txt? | Consider noindex instead |
| Internal search | Are search result URLs being indexed? | Use noindex or selective blocking |
| Testing | Did you inspect URLs in Search Console? | Always test after changes |
This checklist is especially useful for bloggers managing multiple WordPress sites. A single wrong rule can affect thousands of posts, so it is worth checking slowly.
How to Check robots.txt in Google Search Console
If Search Console reports a robots.txt issue, do not panic immediately.
First, ask three questions.
Is this URL supposed to appear in Google search?
If yes, remove the robots.txt block.
Is this URL low-value but harmless?
If yes, the block may be intentional.
Is this URL already indexed and you want it removed?
If yes, do not rely only on robots.txt. Use noindex or other removal methods depending on the situation.
Search Console’s URL Inspection Tool is useful because it shows whether Google can crawl a specific URL and whether crawling is blocked by robots.txt. After changing robots.txt, it may take some time for Google to recrawl and update the status, so do not expect every report to change instantly.
Common robots.txt Mistakes
The first mistake is treating robots.txt as a security tool.
robots.txt is public. Anyone can visit it. If you put sensitive paths inside robots.txt, you may actually reveal where those paths are. Private files, login-only content, customer data, or confidential documents should be protected with authentication, permissions, or server-level security—not robots.txt.
The second mistake is putting noindex inside robots.txt.
This is not a reliable modern Google SEO strategy:
User-agent: *
Noindex: /private/
If you want noindex, use a meta robots tag or HTTP header.
The third mistake is blocking CSS and JavaScript.
Search engines need to render pages. If important theme or plugin files are blocked, Google may not see the page the same way users do.
The fourth mistake is copying someone else’s robots.txt file without understanding it.
A WooCommerce store, a news site, a forum, and a personal blog may need different crawl rules. Copying blindly can break things.
Recommended robots.txt for Kori Think-Style Blogging Sites
For a content-heavy WordPress blog focused on blogging, SEO, Search Console, WordPress operation, and technical guides, I would keep robots.txt clean.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://korithink.com/sitemap_index.xml
Then manage indexing through your SEO plugin.
For example:
- Important posts: index
- Important categories: index
- Thin tag pages: noindex
- Author archives: noindex or index depending on strategy
- Attachment pages: redirect to media file or parent post
- Internal search pages: noindex or block depending on crawl volume
This separates the job of each SEO tool.
robots.txt manages crawling.
noindex manages search appearance.
sitemaps help discovery.
internal links help priority.
Search Console helps diagnosis.
When these pieces work together, WordPress SEO becomes much more stable.
Once you understand robots.txt, you have already learned one important part of technical SEO.
But higher search rankings do not come from robots.txt alone.
Keyword research, title structure, internal links, sitemaps, indexing status, content quality, and page speed all work together to help a blog grow steadily.
For the next step, you may also want to read “SEO Complete Guide: Search Intent, Keywords, and Technical SEO”
That guide goes beyond robots.txt and explains the bigger SEO workflow that bloggers need to understand when they want to build long-term organic traffic.
Kori’s Take
robots.txt is not something to fear, but it is something to respect.
For bloggers, the best robots.txt file is rarely fancy. It is usually short, clean, and intentional.
My final rule is simple:
- Do not block public blog posts.
- Do not block images unless you have a clear reason.
- Do not use robots.txt as a privacy tool.
- Do not use robots.txt as a replacement for noindex.
- Keep your sitemap URL visible.
- Test important URLs after every change.
A blog is built to be discovered.
robots.txt should guide search engines, not confuse them.
When the file is simple and the indexing strategy is clear, Google can spend more time understanding the content that actually matters.
robots.txt Explained Q&A
Q1. Can robots.txt remove a page from Google search results?
Not reliably. robots.txt can stop crawlers from visiting a page, but it does not guarantee that the URL disappears from search results. If you want a page removed from search results, use noindex or another proper removal method.
Q2. Should I block WordPress tag pages in robots.txt?
Usually, no. If you do not want tag pages indexed, it is often better to apply noindex through an SEO plugin. If you block tag pages with robots.txt, search engines may not be able to see the noindex signal.
Q3. Should I include my sitemap in robots.txt?
Yes, it is a good practice. You should still submit your sitemap in Google Search Console, but adding the sitemap URL to robots.txt gives search engines another clear path to discover your site structure.
robots.txt Explained References
This article was written with reference to Google’s official documentation on robots.txt, crawling, indexing, noindex rules, and Search Console page indexing reports.
- Google Search Central, Introduction to robots.txt
- Google Search Central, How Google Interprets the robots.txt Specification
- Google Search Central, Block Search Indexing with noindex
- Google Search Console Help, Page Indexing Report
- Google Search Central, Technical SEO Fundamentals
#robotstxt #WordPressSEO #TechnicalSEO #GoogleSearchConsole #Crawling #Indexing #Sitemap #Noindex #BlogSEO #SearchEngineOptimization
👉 robots.txt Explained Read Next
If this article was helpful, you may also want to read the posts below.
They will help you understand the same topic in a broader and more practical way.
Why Google Indexing Fails: WordPress SEO Guide for Search Console Errors
Search Console Guide for WordPress SEO: Indexing, CTR, and Organic Traffic
Indexing Principle: How Google Finds, Crawls, Stores, and Ranks Your WordPress Blog Posts
If your thoughts feel a little lighter, that’s enough.
See you in the next question — Beautiful KoriThink