Web Crawler
BeginnerA bot that systematically browses the web by following links, discovering and indexing pages — the discovery engine behind search engines and large-scale scraping.
In depth
A web crawler (spider) is a program that traverses the web automatically: it fetches a page, extracts its links, adds them to a queue, and repeats — mapping outward from seed URLs until it has covered its target territory. Googlebot crawling the entire public web is the canonical example; a scraper walking an e-commerce site's category tree is the same idea at smaller scale.
Crawling vs scraping
The terms blur together but name different jobs: crawling is discovery (finding which pages exist), scraping is extraction (pulling structured data out of them). A price-monitoring pipeline crawls category and pagination links to enumerate product URLs, then scrapes each product page for prices and stock. Most real systems do both in one loop.
What well-behaved crawlers do
- Respect robots.txt and crawl-delay hints, honoring site owners' stated rules.
- Identify themselves with an honest user agent (major crawlers also support reverse-DNS verification, since bad bots impersonate Googlebot constantly).
- Throttle politely: spacing requests to avoid degrading the site — both ethics and self-interest, since aggressive crawling triggers rate limits and bans.
- Deduplicate and prioritize: canonicalizing URLs and revisiting important pages more often than stale ones.
Where proxies enter
At scale, crawls distribute across rotating IPs to stay under per-IP rate limits and avoid single-point blocks — one reason commercial crawling infrastructure and rotating proxy pools are inseparable in practice.
Examples
- Googlebot follows links across billions of pages to keep the search index current.
- A price-intelligence crawler walks a retailer's category pages nightly to enumerate product URLs for scraping.
- An SEO tool crawls a client's site to find broken links and orphan pages.
Common use cases
FAQs
Crawling discovers pages by following links; scraping extracts data from those pages. A typical pipeline crawls to enumerate URLs, then scrapes each one for structured data — most production systems combine both.
Crawling publicly accessible pages is broadly lawful in most jurisdictions, with important limits: respect for terms of service, copyright, personal-data rules, and computer-access laws. Courts have generally treated public-data crawling more permissively than authenticated or evasive access.
Sites impose per-IP rate limits, and large crawls from one address get throttled or banned quickly. Distributing requests across a rotating pool keeps per-IP volume low and the crawl inside the site's tolerance — politeness at scale.