We should use robots.txt file to restrict any search engine for crawling. In this video WsCube Tech explain you how to create robots.txt. The Robot Exclusion Standard = Robots Exclusion Protocol = robots.txt Protocol 1. You need a robots.txt file only if your site includes content that you don't want search engines to index like Your site Admin Panel. 2. User-agent: *Disallow: /Admin Panel/ 3. less then bracket meta name="robots" content=" noindex,nofollow“ greater then bracket 4. less then bracket meta name="googlebot“ content="noindex" greater then bracket 5. X-Robots-Tag: noindex Link – www.domain-name.com/robots.txt Follow Us On : Facebook : https://www.facebook.com/wscubetech.india Linked in : https://www.linkedin.com/company/wscube-tech Google+ : https://plus.google.com/+wscubetechjodhpur YouTube : https://www.youtube.com/c/wscubetechjodhpur Twitter : https://twitter.com/wscube
