Robots.txt file is a text file that can be saved to a website’s server.
It determines if and when the search engine crawlers can visit a website’s subpages and include them in their index. In doing this, certain subpages can be excluded from the search results.
For example: by using robots.txt files you can keep a website’s archives from being included in the search results. Some search engines however choose to ignore the robots.txt files. If a subpage needs to be really hidden from search then engines it should be password protected.
Build your Knowledge with our Marketing Packs.
Google Ads (AdWords) Google Keyword Planner Google Knowledge Graph Google Lighthouse Google+-Local (Google Places) Google Maps Google Mobile Updates Google My Business Google News Google Panda Update Google Penguin Update Google Phantom Update Google Places Google Search Console Google Shopping Google Trends Google Updates Overview Google's Hummingbird Algorithm Study
Search Engine Search Engine Advertising Search Engine Guidelines Search Engine Marketing Search Engine Optimization Search Engine Registration Search Engine Spam Search Result Search Term Search Volume SEO Visibility FAQ Site Optimization: Learn how to improve your website Sitemap.xml SSL Encryption Structured Data