I think you’re little bit confused about this post! Really? Oh sorry…Just kidding!!! It’s all about WordPress robots.txt file plugin. But I’m writing as well as others WordPress tutorials. Because most of the people search for a better solution on WordPress robots.tx file. No matter, if you come from Google or WordPress because here i will discuss all about this plugin and will reply for your WordPress robots.txt issues. Don’t worry anymore – you’ll learn more about robots.txt file, importance of robots.txt file for better SEO performance and using the WordPress Robots.txt File plugin.
What is Robots.txt File?
A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers. The file uses the Robots Exclusion Standard, which is a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers.
You only need a robots.txt file if your site includes content that you don’t want Google or other search engines to index.
Why Robots.txt File Important?
Do you need a robots.txt file and what is the importance of it? Your site’s robots.txt file play an important role in your site’s overall SEO performance. It basically allows you to communicate with search engines and let them know which parts of your site they should index.
Really Need a WordPress Robots.txt File?
Missing of a robots.txt file will not stop search engines from crawling and indexing your website. However, it is highly recommended that you create one. If you want to submit your site’s XML sitemap to search engines, then this is where search engines will look for your XML sitemap unless you have specified it in Google Webmaster Tools.
How to Use WordPress Robots.txt File Plugin?
Robots Txt File plugin allows you to create or edit the content of your robots.txt file. It’s really simple to use. Just install the plugin by using dashboard or upload “robots-txt-file” folder to your plugins directory. Now activate it from WordPress admin panel and go to Dashboard => Settings => Reading. You’ll find a box with title Robots.txt File. You can write and edit the default robots.txt content in the box. And click the “Save Changes” button to save your edited file. All done!
Conclusion
I hope you’re using the benefit of robots.txt file plugin and liked it. However, If you have any issue don’t be late to ask me by comment box. And finally you think that it’s useful for others also then don’t forget to hit on share buttons.
Disallow: /website url
so this is right or not please reply me fast
This is not right.
Ok i got the right way
User-agent: *
Disallow: /
This is right ??
Yes, that’s right!
Thank you :)
How to block me website all content by robots.txt file
I don’t understand what you mean? Please explain it clearly..
Hi there,
My site “sportskanazee.com” is not indexed by google and whenever i’m searching for my website on google, it throws a message i.e.
A description for this result is not available because of this site’s robots.txt
Though i have unchecked the box on “reading” i.e. Discourage search engines from indexing this site on wordpress, still after one week, the message i get on google is “A description for this result is not available because of this site’s robots.txt”
Kindly help solving this issue so that my website is indexed on google and my site will get ranking.
Thanks n Regards
Prem
Hey,
I understand your issue. Please kindly contact with me via live chat on bottom right tab. I will give you best solution on it.
Thanks!
I changed the content of robots,txt file and acces http:/url/robots.txt.It is showing old text.Not new.
Please send me the link then i can help you. Thanks!
My web site was recently hacked. The robots.txt file was added during this hack. Robots does not show in my plug ins How do I get rid of it? It seems as if many malware attempts are being made to access this page
Are you using my plugin? If so you should configure it to rejects malware and boots. Please can you buy pro support and send me a private email using contact. Thanks!
Why did my comment is not shown . I have completed all this applicationclearly but failed to fit robottxt file.
What’s the issue? Can you explain me details please?
hi
used robots.txt in wordpress http:// its working fine, but robots.txt in wordpress https:// not workings show error
404 not found can you help on this.
Sorry to hear about that issue! Please can you email me details about it. It will be better to assist you. Thanks!
Hi,
I was told by google to edit the robots.txt file so that they can crawl it but I don’t know what to edit. This is what I see.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
That’s all I see. What should I delete or add? Thanks for your advice.
Anne.
same thing for me
i want Google to search my entire site
Anne & Joe, Please see the sample how you can do this:
User-Agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Sitemap: http://www.example.com/sitemap.xml
Thank You!
Hi, I am trying to deactivate the robots.txt file so I can run Screaming Frog Application. It keeps telling me my website is blocked. I have tried changing the code in your plug-in to without success.
User-agent: *
Disallow:
I have tried deactivating the plugin – no success
I have gone to my hosting file manager and cannot seem to find the robots.txt file there.
Could you please advise how I can short term disable the plugin? Greatly appreciated
Do you want to deactivate this plugin? Or want to remove any other? If you want to remove this one then just go to active plugins from dashboard then deactivate it. Thanks!
Hey!
I used your plugin to make some changes but the changes aren’t being reflected on the live robots.txt file. I cleared the cache on my browser and in the WP caching plugin, many times – but it doesn’t help. Any suggestions? Thanks!
morgan
If you use double way to run robot txt on blog then it will be sometime problem. So please try with this one and deactivate another one. Hope it will work fine there. Thanks!
well, that is working for my site, but if i need to configure my robot.txt file for Google webmaster tools, this plugin do not offer me an option to put a url suppose http://www.mysite.com/robot.txt . can you plz update that how to make it viable for google, so it access it and find my site robots.txt information. thanks
Regards,
Tariq
why you need to visible this? It’s an security issue if you make it visible. And normally all others using it as non-visible content. But only google can read this.
There came a robots.txt file included in the theme I installed. It’s very long, disallowing a whole list of things. I haven’t added anything, it just was there when I changed the theme. I want to allow crawling now. This plugin says it can edit an existing robots.txt file, but is this then not true?
It’s editing existing robots.txt file but i said to use single way. But no worry the new robots.txt is using from this plugin. If the old one is keep there that’s inactive. Okay
Hi, I installed the Robots.txt File plugin to edit my robots file. But when I go to Settings > Reading, it does not show my currect Robots file, but the default. My current one, which I want to edit because it prevents some crawling I need to allow, is much longer. If I edit this one in Settings > Reading and save it, nothing seems to happen, it’s like the existing one stays there anyway. Or is there a delay?
If you use one more way to add the robots.txt then it will be problem. You can add the text in Settings > Reading Robots.txt box. Then remove that content from all others way. And there is no delay for it. If you still having this issue let me know. Thanks!