Most JavaScript scrapers are slower and more fragile than they need to be, and the cause is usually one decision made too early: reaching for a headless browser before checking whether the page needs one.
The check takes about ten seconds. It also decides your dependency list, your memory footprint, and how often the thing wakes you up at 3am.
The Question To Ask Before You Write Any Code
Request the page the way a plain HTTP client would, and look at what actually comes back.
curl -s https://example.com/products | grep -i "price"
If the values you want are sitting in that response, the page is static as far as your scraper is concerned. The server did the rendering and handed you finished HTML. If you get back a shell of empty divs and a script bundle, the page assembles itself in the browser, and you need something that can execute JavaScript.
There is a third case worth checking before you commit to either path. Open the network tab and watch what the page requests after it loads. A lot of sites with no public API pull their content from a JSON endpoint that you can call directly. That route is faster than both alternatives and it tends to survive redesigns, because the markup can change completely while the endpoint stays where it is.
The Static Path: Fetch And Cheerio
For a static page the whole job is one HTTP request and a parser. Node's built in fetch gets the HTML, and Cheerio gives you a familiar selector API over it without any of the weight of a real DOM.
const res = await fetch("https://example.com/products", {
headers: { "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" }
});
const $ = cheerio.load(await res.text());
const items = $(".product-card").map((i, el) => ({
name: $(el).find("h2").text().trim(),
price: $(el).find(".price").text().trim()
})).get();
This runs in milliseconds, uses a few megabytes, and happily handles hundreds of concurrent requests on a single Node process, because the event loop is doing nothing but waiting on sockets. Nothing about a headless browser improves this outcome.
The Dynamic Path: When The Browser Is The Only Way
When content only exists after render, Playwright or Puppeteer stops being overhead and starts being the point. Three habits keep that path from becoming the flaky part of your system.
Wait for a signal, not for a duration. A fixed sleep is a guess that is either too slow or too short. Waiting for the selector you actually need is neither.
Block what you do not read. Images, fonts and media are most of the page weight and none of the data. Aborting those requests often cuts page time by more than half.
Reuse one browser, many contexts. Launching Chromium per URL is the single most expensive mistake in a dynamic scraper. One browser instance with a fresh context per job gives you isolation without the startup cost.
The decision tree behind all of this, along with the libraries for each path, is laid out in more depth in this guide to web scraping with JavaScript.
What Breaks In Production
Scrapers rarely fail on the day you write them. They fail six weeks later, and almost always for one of four reasons.
Selectors that were too specific. div > div:nth-child(3) > span encodes the entire layout. Anchor on text, ids, or data attributes instead, and your scraper survives a redesign that moves things around.
Retry loops with no ceiling. A failing target plus an eager retry is how a polite scraper turns into something that looks like an attack. Cap the attempts, back off exponentially, and log the failure rather than hammering through it.
Concurrency set by wishful thinking. Node will happily open a thousand sockets. The site on the other end will happily stop answering. Pick a concurrency you would be comfortable defending, then add delays between requests.
No schema check on the way out. If a field silently starts coming back empty, you want to know that day, not when someone queries the table a month later and finds a column of nulls.
The Takeaway
The library is not the interesting decision. Whether the page needs a browser at all is the interesting decision, and it is answerable before you write a line of code. Check the raw response first, look for the JSON endpoint second, and reach for the browser third, when the page has genuinely earned it.
Top comments (0)