Crawlers

A crawler is a program used by search engines to collect data from the internet.

When a crawler visits a website, it picks over the entire website’s content (i.e. the text) and stores it in a databank. It also stores all the external and internal links to the website. The crawler will visit the stored links at a later point in time, which is how it moves from one website to the next. By this process the crawler captures and indexes every website that has links to at least one other website.

Build your Knowledge with our Marketing Packs.

Core Web Vitals
Improve your website performance through UX Optimizations
27 resources
Searchmetrics Study: Online Retail Market Analysis 2019
Online Marketing: proven strategies and solutions for businesses
27 resources
Google Updates: Latest News and Analysis
How do Google updates affect business rankings and revenue?
23 resources
Searchmetrics Study: Niche Ranking Factors
Grow your pipeline and increase revenue through organic search and demand optimization
47 resources
A
Accelerated Mobile Pages Agile Content Development Alt Attributes Anchor Text
B
Backlinks Bad Neighborhood Blockers Briefing Broken Links Business directory
C
Cloaking Competition Content Editor Content Gap Analysis Content Marketing Content Relevance Content Score Conversion Cookies Crawlers
D
Defective Links Domain Popularity Domain Trust Duplicate Content
F
Featured Snippets Frame
G
Google Ads (AdWords) Google Keyword Planner Google Knowledge Graph Google Lighthouse Google+-Local (Google Places) Google Maps Google Mobile Updates Google My Business Google News Google Panda Update Google Penguin Update Google Phantom Update Google Places Google Search Console Google Shopping Google Trends Google Updates Overview Google's Hummingbird Algorithm Study
H
Homepage
I
Index Internal links
K
Keyword Keyword Cannibalization Keyword Density Keyword Proximity Keyword Stuffing
L
Link Juice Link Popularity Link text
M
Meta Description Meta Search Term Meta Tag Meta Title
O
Offpage Optimization Onpage Optimization
P
Page Page Content Page Title Paid Listing Plagiarism Finder Progressive Web Apps
R
Ranking Factor Ranking Opportunities Rankings Rich Snippets Robots.txt
S
Search Engine Search Engine Advertising Search Engine Guidelines Search Engine Marketing Search Engine Optimization Search Engine Registration Search Engine Spam Search Result Search Term Search Volume SEO Visibility FAQ Site Optimization: Learn how to improve your website Sitemap.xml SSL Encryption Structured Data
T
Topic Explorer Topical Relevance
U
Universal Search URL User Experience User/Search Intent User Signals
W
Web Catalogues Web Directory Web Pages Webmaster Guidelines Website Website Structure
Z
Zero Click Searches