Robot.txt is a Meta tag element used to prevent robot from getting access to your web content. Webmasters can use robot.txt to disallow other software on the internet form access to either the whole of the server other part of it.
Does it hide content on website, this Meta tag element does not hide any data from your site visitors, it is only design to prevent robot (mostly search engines from getting an access from your website.). Human visitors can see all your post and pages, therefore if you intend to hide any data from humans don’t use this robot.txt. the untold money making secret
Does it prevent malwares and viruses from accessing website, robot.txt cannot stop malwares sent by hackers to operate on your website, as this malwares and virus can break the security of website.
This Meta robot element can therefore only prevent search engines from crawling and indexing your website, thereby reducing its search engine visibility.
Most blogger publishers do not know that this thing exist, , this is why they will continue to see low search traffics to their blog, by default, Blogger (BlogSpot) disallow this meta tag file for both header and text files,. Then what should you do? The best thing is to remove the robot element to increase your search engine visibility, to get your blog crawled and indexed for next searches. guide to picking optimized domain
How do you allow it? Go to your blogger dashboard, locate the setting tab and scroll down to search preference, edit under the crawling and indexing the custom robot.txt and custom robots header tag from disallow to enabled. Click on the yes and leave text box blank and for the custom robots header tag, click on the yes and tick the all button. That’s all,
How do I come across it? I was trying to submit my blog to a search engines through scrubtheweb and I run a free meta tag analyzer and I notice this errors, I saw my search visibility was 39 but when I do the above editing , my search visibility changes from 39 to 57, wonderful. how to add meta tag for blogger