Headless Browser
IntermediateA real browser running without a visible window — full JavaScript and rendering for automation, but with detectable traces that anti-bot systems hunt for.
In depth
A headless browser is a genuine browser engine — usually Chromium or Firefox — running without displaying anything on screen. It parses HTML, executes JavaScript, renders layouts, and fires events exactly like a normal browser; it simply has no window. That makes it the workhorse of modern scraping and testing: driven by Puppeteer, Playwright, or Selenium, it can operate sites that plain HTTP libraries can't touch because the content only exists after JavaScript runs.
Why scrapers need it
- JavaScript-built pages: single-page applications assemble their content client-side; an HTTP library sees an empty shell, a headless browser sees the finished page.
- Real interaction: clicking, scrolling, form-filling, and waiting for dynamic content — indispensable for flows that require behaving like a user.
- Authentic protocol behavior: a real browser produces genuine TLS handshakes and header patterns, clearing hurdles that expose raw HTTP clients instantly.
The detection cat-and-mouse
Headless operation leaves traces, and anti-bot vendors catalogue them: the navigator.webdriver flag, missing plugin and codec inventories, headless-typical GPU strings, absent behavioral noise (no mouse jitter, superhuman timing), and subtle rendering differences. Stealth plugins patch the obvious tells; antidetect browsers go further by substituting full coherent fingerprints. Running "headful" (with a window, possibly on a virtual display) closes some gaps at the cost of resources.
Cost note
Each headless instance is a full browser eating hundreds of megabytes of RAM — orders of magnitude heavier than an HTTP request. Use plain requests where the site allows; reserve browsers for targets that demand them.
Examples
- A Playwright script drives headless Chromium through an infinite-scroll feed, capturing content a plain HTTP client never sees.
- An anti-bot script reads navigator.webdriver and headless-typical GPU strings, then blocks the session.
- A CI pipeline runs a site's test suite in headless browsers on every commit.
Common use cases
FAQs
Often, yes — through automation flags like navigator.webdriver, missing plugins and codecs, headless-typical WebGL strings, and the absence of human behavioral noise. Stealth patches and antidetect browsers close these gaps to varying degrees; it's a continuous arms race.
Use plain HTTP requests when the data is in the initial HTML or a callable API — they're vastly cheaper. Use a headless browser when content is built by JavaScript, requires interaction, or sits behind defenses that demand authentic browser behavior.
Playwright and Puppeteer are the modern defaults — fast, well-documented, with strong waiting and interception APIs. Selenium remains standard where legacy infrastructure or broad language support matters. All three drive the same underlying browsers.