Sorry, we don't support your browser. Install a modern browser
Hide from search engines should use noindex, nofollow instead of robots.txt#590
When an admin enables hide from search engines, it modifies the robots.txt for that subdomain/domain. This is incorrect as the page can still appear in search results if other pages link to it.
Remove the robots.txt rule and instead use a meta tag or a response header as outlined in the documentation:
https://developers.google.com/search/docs/crawling-indexing/block-indexing
a year ago
2
Changed the status to
In Progress
a year ago
@Amit from Nolt if possible, please provide the option to do either noindex or robots.txt. There are cases where you want to do one or the other.
https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
https://www.reddit.com/r/SEO/comments/ycxae5/comment/itqhf3v
a year ago
1