One of my extension's features imports a whole YouTube playlist into your notebook in one click — paste a playlist link, it pulls every video in as a source. I pointed a 124-video playlist at it to test. It imported 15.
Not 124. Fifteen.
That kicked off two days of chasing round numbers, and every round number turned out to be a lie told by someone else's system.
- My importer was reading YouTube's playlist RSS feed — clean, official, no scraping. Turns out that feed hard-caps at ~15 items no matter how big the playlist is. 15 wasn't my data. It was YouTube's feed limit wearing my data's clothes.
So I switched to scraping the playlist page's embedded JSON (ytInitialData) — the same blob the page itself renders from. First run after the switch: 0 videos. The structure had shifted under me. The old playlistVideoRenderer.videoId path was gone; today's YouTube wraps each video in a lockupViewModel with the id sitting in contentId. Fixed the path.
- Now it pulled 100. Progress! But the playlist had 124, and 100 is exactly one page. YouTube paginates: to get page 2 you send back a "continuation token" you find at the bottom of page 1. I was finding the token — I could log it — but my loop behaved like there was none.
This one was mine, and it's a good one. My tree-walker returned a single token as it recursed: walk each child, if a child returns a token, keep it. The problem is that after finding the real token, the walk kept going into sibling branches that returned nothing — and "nothing" overwrote my good token with null. The right answer was there for a moment, then a later, emptier branch clobbered it. A DFS that should have been "first non-null wins" was quietly doing "last write wins." Fix: stop returning one token; push every token into an array and take the first. Instantly: 124.
Except — not from inside the actual extension. From a standalone console test, 124. From the extension, still 100.
- My panel runs in an iframe on the notebook page. When it fetched YouTube's pagination endpoint, the request carried Origin: chrome-extension://… — and YouTube's API takes one look at that and returns 403. The page fetch (a plain GET of HTML) worked fine; the API call (the one thing that gets you past 100) didn't. I only caught it because I finally logged the proxy's status codes: GET 200, POST 403.
Fix: move the fetches into the background service worker, and add a declarativeNetRequest rule that rewrites Origin/Referer to https://www.youtube.com for that endpoint. Now the request looks like it came from YouTube itself. 200. 124 videos. Done.
The thing I keep thinking about: 15 and 100 were both round numbers, and both were somebody else's limit masquerading as my result. When a scraper stops at a suspiciously clean number, that's rarely where your data ends — it's where a page size, a feed cap, or a rate limit begins. The number is a fingerprint of the wall you just hit, not of the thing you're counting.
That's the tax on building a tool that lives on top of someone else's product with no API contract. Every layer — the feed, the DOM shape, the pagination, the CORS policy — can drift or bite, and none of it is yours to stabilize. You just get good at reading round numbers as clues.
What's the most misleading "round number" bug you've hit — where the value looked like an answer but was actually a limit?
— building NotebookBloom in public, #16
Top comments (9)
Dealing with YouTube API pagination and rate limits can be incredibly frustrating when building batch importers. I remember hitting a wall with a similar scraper where the API would silently drop requests if you exceeded the quota, forcing me to implement exponential backoff and chunked processing. Did you end up using the standard YouTube Data API v3 with page tokens, or did you have to scrape the initial page payload directly to bypass those strict quota limits? Handling the state for partial imports is a great edge case to solve for user experience.
Yeah, this is exactly where it hurt. Short answer: I skipped Data API v3 entirely.
The dealbreaker was the API key. It's a client-side extension, so any key I ship gets shared across every single user, and playlistItems.list eats quota fast. One power user importing a few big playlists and everyone else is locked out for the rest of the day. Dead on arrival.
So I went the scrape route. Pull ytInitialData off the playlist page for the first batch, then for pagination I hit YouTube's own internal endpoint (youtubei/v1/browse) with the continuation token from the bottom of each page — same InnerTube API their web player uses. No key, no quota to blow through. The tradeoff is the obvious one: it's undocumented and it drifts. They literally moved the videoId field on me halfway through the project.
On the silent-drop thing you mentioned — InnerTube doesn't have a hard published quota, but it'll soft-throttle if you hammer it, so I space the continuation calls out instead of firing them all at once. Haven't needed full exponential backoff yet, but I can see it coming the day someone points a 2,000-video playlist at it.
And 100% agreed on partial imports — that was the part I cared most about. It dedupes on re-run, so if it dies at video 80 you just run it again and it fills in the rest instead of starting from scratch. Failed items get surfaced with a count, never silently dropped. Silent drops are the cardinal UX sin here.
Bypassing the Data API entirely makes total sense for a client-side extension, since a shared quota is basically a ticking time bomb. Switching to scraping solves the immediate rate limit issue, but I'm curious how you handle the inevitable DOM changes or IP throttling from YouTube. Are you routing the scrape requests through a proxy network, or keeping it strictly local to the user's machine?
Strictly local, and honestly that's the part I'd defend hardest — no proxy, no relay, nothing of mine sits in the path. The requests fire from the user's own browser with their own YouTube cookies and their own IP. From YouTube's side it's indistinguishable from that person scrolling the playlist page themselves, because functionally that's what it is. The only sleight of hand is a declarativeNetRequest rule that rewrites the Origin/Referer header back to youtube.com — a request coming out of a chrome-extension:// origin gets a 403 otherwise. That's it. No server of mine ever touches YouTube.
Which also means there's no shared IP to throttle. This was the whole reason the API key was a non-starter — one shared key, one shared quota, one power user nukes it for everybody. Going local flips that completely: every user brings their own rate budget. Ten thousand users don't stack up against one bucket, they're ten thousand separate buckets. A proxy network would have quietly re-introduced the exact single-choke-point I was running away from, plus a hosting bill I don't want and a privacy story I couldn't stand behind. Hard pass.
The DOM drift you're right about — that's the standing tax and there's no clever way out, only two dampers. First, I don't scrape rendered HTML, I read ytInitialData and the InnerTube JSON, which drifts way slower than the visual DOM — though it still drifts, they moved the videoId field on me mid-project. Second, when the primary path returns nothing, it falls back to the official playlist RSS feed. RSS caps around 15 videos so it's a worse result, but "you got 15 instead of 124" degrades a lot more gracefully than a hard zero while I ship a fix. And the fix, when a field moves, is a one-line selector patch, not a re-architecture. It's a parasite product by nature — living on someone else's platform means budgeting for the day they redecorate. I priced that in from day one.
Relying entirely on the user's own session cookies and IP is a smart way to sidestep API quotas while keeping the architecture completely trustless. Using declarativeNetRequest to mimic natural scrolling is a clever workaround that keeps bot-detection happy without needing a backend. Out of curiosity, how does the extension handle the progressive loading chunks, like the initial 15 videos versus the final 124?
Ha, good question — but the 15 and the 124 aren't two stages of the same load. They're two completely different code paths, and which one you get depends on where the data comes from.
The 15 is the fallback. If the main path can't get a clean read on the page — playlist's not fully public, layout's doing something weird, whatever — I drop down to YouTube's RSS feed for that playlist. The feed is dead simple and never breaks, but it's hard-capped at 15 items on YouTube's end. So if you ever see exactly 15 come back, that's the tell that it fell back to RSS and you're only getting a slice.
The 124 is the real path. First chunk (~100) comes straight out of ytInitialData that's already embedded in the playlist page — no request needed, it's just sitting there in the HTML. Then for anything past that, YouTube itself paginates with a continuation token tucked at the bottom of each response. So I grab that token and POST it to youtubei/v1/browse (their own InnerTube endpoint), get the next ~100 plus the next token, and just loop until there's no token left. That's it — no page numbers, no "give me page 2," you follow the breadcrumb until it runs out.
The one gotcha that bit me there: the continuation token isn't always a single clean field. My first tree-walker did "last write wins" and happily overwrote a good token with a null further down the tree, so pagination just... stopped early. Fix was to collect every token I find into an array and take the first valid one instead of trusting the last. That's the difference between capping at 100 and actually getting to 124.
And each chunk gets appended + deduped as it lands, not at the end — so if it dies mid-loop, the resume picks up from what's already in instead of refetching. The progress you see counting up is literally each continuation page landing.
tl;dr: 15 = degraded RSS fallback, 124 = ytInitialData seed + InnerTube continuation loop. Not the same source getting bigger — one's plan B.
Falling back to the RSS feed when the primary extraction path fails is a solid graceful degradation strategy, especially since YouTube's feeds notoriously cap out at 15 items. It guarantees you at least get partial metadata instead of a hard failure when the layout changes or access gets restricted. Since the RSS feed is inherently limited, do you implement any pagination tricks or secondary fallbacks to eventually catch the remaining 109 videos, or does the user have to manually intervene?
Falling back to the RSS feed is a clever graceful degradation strategy when the primary scraping path hits a wall. Since RSS feeds often cap out at a limited number of items, I am curious how your main code path handles the pagination required to actually fetch all 124 videos. Do you rely on a headless browser for that main path, or are you reverse-engineering the internal API endpoints?
Reverse-engineering the internal API, no headless browser anywhere. A headless browser isn't even on the table for an MV3 extension — I've got no background page that can drive one, and spinning up a whole second browser to read a list I'm already looking at would be absurd overhead.
The one thing I'd push back on is the word "reverse-engineering" — makes it sound sneakier than it is. youtubei/v1/browse is the exact endpoint YouTube's own page hits to load more videos as you scroll. I'm just calling it directly with the continuation token they hand me, from a fetch running in the page's own context, so it's the user's live session — same cookies, same IP, no automation pretending to be a person. It's less "I cracked their API" and more "I stopped waiting for the scroll and asked for the next page myself." The whole loop is that one endpoint until the token runs dry.