A crawler is a program used by search engines to collect data from the internet.
When a crawler visits a website, it picks over the entire website’s content (i.e. the text) and stores it in a databank. It also stores all the external and internal links to the website. The crawler will visit the stored links at a later point in time, which is how it moves from one website to the next. By this process the crawler captures and indexes every website that has links to at least one other website.
Build your Knowledge with our Marketing Packs.
Google Ads (AdWords) Google Keyword Planner Google Knowledge Graph Google Lighthouse Google+-Local (Google Places) Google Maps Google Mobile Updates Google My Business Google News Google Panda Update Google Penguin Update Google Phantom Update Google Places Google Search Console Google Shopping Google Trends Google Updates Overview Google's Hummingbird Algorithm Study
Search Engine Search Engine Advertising Search Engine Guidelines Search Engine Marketing Search Engine Optimization Search Engine Registration Search Engine Spam Search Result Search Term Search Volume SEO Visibility FAQ Site Optimization: Learn how to improve your website Sitemap.xml SSL Encryption Structured Data