.Google's John Mueller answered a question about why Google.com marks web pages that are actually prohibited from creeping by robots.txt and why the it's risk-free to dismiss the similar Search Console documents regarding those crawls.Robot Visitor Traffic To Question Specification URLs.The person talking to the concern documented that robots were producing hyperlinks to non-existent concern guideline Links (? q= xyz) to webpages along with noindex meta tags that are actually also obstructed in robots.txt. What urged the concern is that Google.com is actually crawling the hyperlinks to those webpages, receiving obstructed through robots.txt (without noticing a noindex robots meta tag) after that getting shown up in Google Explore Console as "Indexed, though blocked through robots.txt.".The individual inquired the adhering to inquiry:." But listed here's the major concern: why will Google index web pages when they can not even find the material? What is actually the advantage during that?".Google.com's John Mueller validated that if they can not creep the page they can't view the noindex meta tag. He also produces an appealing mention of the website: search operator, suggesting to dismiss the outcomes because the "typical" consumers won't find those end results.He created:." Yes, you're appropriate: if we can't creep the page, we can't view the noindex. That said, if our company can not crawl the web pages, after that there is actually not a whole lot for us to mark. Therefore while you might view some of those webpages with a targeted web site:- concern, the common consumer won't view them, so I definitely would not bother it. Noindex is additionally alright (without robots.txt disallow), it just means the URLs will certainly find yourself being actually crept (and also wind up in the Explore Console record for crawled/not recorded-- neither of these conditions cause problems to the rest of the web site). The fundamental part is actually that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the restrictions in using the Internet site: search evolved hunt operator for analysis causes. One of those explanations is actually because it is actually not linked to the regular hunt mark, it is actually a different point altogether.Google's John Mueller talked about the site hunt driver in 2021:." The brief solution is that an internet site: query is actually certainly not meant to be complete, neither utilized for diagnostics purposes.A website inquiry is a particular type of hunt that confines the end results to a certain website. It's essentially only the word internet site, a bowel, and then the site's domain.This concern restricts the end results to a details internet site. It's not suggested to become a complete collection of all the webpages coming from that internet site.".2. Noindex tag without using a robots.txt is actually alright for these sort of circumstances where a crawler is connecting to non-existent webpages that are obtaining found out by Googlebot.3. URLs with the noindex tag are going to create a "crawled/not catalogued" item in Search Console and that those will not have a negative result on the remainder of the website.Review the inquiry and also address on LinkedIn:.Why would Google.com index pages when they can't also find the web content?Featured Graphic through Shutterstock/Krakenimages. com.