Robots.txt
Robots.txt file is a text file that can be saved to a website’s server.
It determines if and when the search engine crawlers can visit a website’s subpages and include them in their index. In doing this, certain subpages can be excluded from the search results.
For example: by using robots.txt files you can keep a website’s archives from being included in the search results. Some search engines however choose to ignore the robots.txt files. If a subpage needs to be really hidden from search then engines it should be password protected.
Build your Knowledge with our Marketing Packs.
C
G
Google Ads (AdWords)
Google Keyword Planner
Google Knowledge Graph
Google Lighthouse
Google+-Local (Google Places)
Google Maps
Google Mobile Updates
Google My Business
Google News
Google Panda Update
Google Penguin Update
Google Phantom Update
Google Places
Google Search Console
Google Shopping
Google Trends
Google Updates Overview
Google's Hummingbird Algorithm Study
H
S
Search Engine
Search Engine Advertising
Search Engine Guidelines
Search Engine Marketing
Search Engine Optimization
Search Engine Registration
Search Engine Spam
Search Result
Search Term
Search Volume
SEO Visibility FAQ
Site Optimization: Learn how to improve your website
Sitemap.xml
SSL Encryption
Structured Data