How to Scrape YouTube: Search, Videos, Comments, and Channels

YouTube takes in more than 500 hours of video every minute and serves it back to a base of 2.49 billion monthly active users. It works fine in a browser. The problem starts when we try to fetch a video URL with requests: we get back a 200, but the HTML has zero search results, zero comments, and a full-screen consent overlay where the page should be.

That is actually good news. Once past the consent wall, every piece of data on every YouTube surface lives in a place we can reach with one HTTP call. No headless browser. No JavaScript engine. The win condition is simple: turn the consent wall into structured rows of search results, video metadata, comments, and channel data.

We will route every request through Scrape.do with super=true, pull ytInitialData and ytInitialPlayerResponse straight from the page HTML, then page through InnerTube continuation tokens for everything that loads after the first screen. Four clean outputs: a CSV of search results, a JSON record per video, a JSON tree of comments with nested replies, and a JSON file per channel with paginated video lists.

Full working code on GitHub ⚙

If you'd rather skip InnerTube parsing entirely, Scrape.do offers a dedicated YouTube API. One HTTP call returns the full search page as structured JSON: videos, channels, playlists, Shorts shelves, and ads, with sp tokens for sort, duration, 4K/HD/LIVE/CC, upload date, and result type. Jump to the Plugin API section for examples.

Challenges with Scraping YouTube

YouTube's anti-bot is aggressive but probabilistic. From a datacenter IP (cloud VMs, VPS, shared hosting), a clean Python request almost always lands on a bot-detection consent wall: HTTP 200, HTML that looks like a real page, but with none of the data we wanted. From a residential IP the same request may succeed today and fail tomorrow.

This is the kind of target we like. The failure mode is predictable when it fires, the data is already structured once we get past the gate, and Scrape.do absorbs the gate so we can stick with requests instead of building our own anti-detection stack.

YouTube fingerprints every request against IP reputation, TLS handshake, browser fingerprint, and JavaScript challenge solving. Datacenter IPs fail every check. Residential IPs pass most of them, but any flag we trip (rapid requests, suspicious User-Agent, missing cookie) still lands us on a 200 with a consent wall instead of a 403. The fix is to route every request through Scrape.do with super=true, which sends each request from a residential IP with a real browser fingerprint and a challenge solver attached.

Every request needs the parameter. The initial GET, every continuation POST, every reply continuation POST. Skip it and we are back to rolling dice with YouTube's anti-bot on every call.

Two Data Sources: Embedded JSON vs. InnerTube API

Two JavaScript variables in the page HTML carry almost every field we need. ytInitialData holds the rendered page state. ytInitialPlayerResponse holds the player-specific fields on watch pages.

We pull both out of the HTML the same way:

Nothing fancy. Two lines, both blobs out. We will reuse this pattern under video (twice, once per blob), channel, and comments.

Anything that loads after the first page comes from InnerTube, YouTube's internal POST API. The endpoint name changes per surface: /youtubei/v1/search, /youtubei/v1/browse, /youtubei/v1/next. The body shape stays the same on all three: a context.client block plus a continuation token from the previous page.

Two more things to grab from the initial HTML before we can call InnerTube: the embedded API key and the WEB client version. Both are one-line regex matches against the page source. The client version rolls forward roughly once a week with YouTube's frontend deploys, so we read it fresh on every run.

Scraping YouTube Search Results

Search results come first because the page exposes every pattern we will lean on later: the embedded blob extract, the WEB client version regex, the InnerTube continuation loop.

A search like web scraping python returns about 18 organic videos on the first page, plus zero or more "Shorts shelves" interleaved through the results. Each shelf groups 10 to 30 short-form videos in a different schema than regular video results, so we dispatch on item type as we walk the list.

The Quick Way

If you don't want to maintain the InnerTube parsing yourself, the /plugin/google/youtube endpoint returns the same data as one HTTP call. Each request costs 10 credits.

The endpoint takes a search_query plus an optional sp token. sp doubles as a filter and pagination cursor: short values control sort (CAM== view count, CAI== upload date), result type (EgIQAg== channels only, EgIQAw== playlists only), duration (EgIYAg== over 20min), features (EgJwAQ== 4K, EgJAAQ== LIVE), and upload date (EgIIAg== today, EgIIAw== this week). Long values are continuation tokens from the previous response's pagination.next_page_token. The plugin routes both automatically.

Two clean wins for this use case: views come back parsed as integers ("1.2M views" → 1200000), and channel.verified, extensions: ["4K"], and live: true/false are already extracted on every entry. No runs array walking, no view-string regex, no continuation-token bookkeeping.

The rest of this article walks through doing it manually: same data, fewer abstractions, helpful when you want to learn the InnerTube model or extract fields the plugin doesn't expose.

Prerequisites

One library:

No BeautifulSoup, no headless browser. Every parsing step works against JSON blobs.

The other prerequisite is a Scrape.do account. Sign up at scrape.do for a free account that includes 1,000 monthly requests, which is enough to run everything we cover here on multiple targets. The dashboard shows the API token at the top of the home page. We will paste it into a TOKEN constant at the top of every code block.

Imports and the config we will tweak per run:

We need a fetch helper that wraps every Scrape.do call, so one URL pattern handles both GETs and POSTs:

The customMethod=POST parameter is the one quirk: Scrape.do needs it to forward the request method correctly to YouTube. We will hit this fetch helper again when we get to video, comments, and channel.

Parsing ytInitialData for Videos and Shorts

The search-results page response includes a <script> tag with var ytInitialData = {...}; near the bottom of the document. Every field on the page sits inside that blob. We pull it out with the two-line pattern from earlier:

The result list lives a few levels deep inside data.contents. Each entry is either an itemSectionRenderer (containing the actual results) or a continuationItemRenderer (carrying the next-page token).

Inside the itemSectionRenderer.contents, items are tagged by type. A videoRenderer is a regular video result. A gridShelfViewModel containing nested shortsLockupViewModel items is a Shorts shelf. A third type, lockupViewModel, appears for playlists or channels and we skip it.

We write a small parser for regular videos:

The title and channel name come back as runs arrays because YouTube splits strings into typed segments when they include hyperlinks or bold formatting. We rebuild the full string by joining the runs. The thumbnail at index [-1] is the highest-resolution variant YouTube ships.

Shorts use a different schema. The shelf hands us only video ID, title, view count, and thumbnail. Channel name, duration, and upload date are not exposed for Shorts on the search page, only on the individual short's watch page. We write the parser around those limitations:

Both row shapes share the same column set so a single CSV holds videos and Shorts side by side. Fields the search page does not provide for Shorts come out as None, and a type column lets us split them later.

We add a dispatcher that walks the section list and hands each item to the right parser:

That is all the dispatching we need, routing three renderer types to two parsers. The position counter threads through the loop so page-2 results start where page-1 left off, and the continuation token at the end is what we need for the next page.

InnerTube Continuation Pagination

After the first 18 organic results, more pages come from POST https://www.youtube.com/youtubei/v1/search. The body asks for two things: a context.client block (using the WEB client version we extracted earlier) telling YouTube which web client we are pretending to be, and a continuation token from the previous page.

If the loop suddenly returns empty pages even though tokens are still coming back, the client version is the first thing to check. It looks like 2.20260416.01.00 and rolls forward roughly once per week with YouTube's frontend deploys.

We loop until the row count hits the cap or the continuation token comes back empty:

Each response ships the next batch of items inside appendContinuationItemsAction.continuationItems, in the same renderer shape as the initial page. We feed them through the same extract_items we used the first time. If a page comes back empty even though the token was set, we stop early.

Export to CSV

Every row, video or short, has the same column set, so we write everything out with DictWriter using the first row's keys as the header:

A run with SEARCH_QUERY = "web scraping python" and MAX_RESULTS = 60 produces a CSV with 60 rows split between regular videos (full metadata) and Shorts (with None in the columns the search page does not populate for short-form content). Two pages of pagination, ~37 items on the first page and ~36 on the second, sliced to exactly 60.

The search page stops being a page and becomes 60 rows of CSV.

Scraping YouTube Video Metadata

Single-video metadata is the simplest case: one HTTP call, every field in the same response, no pagination.

The catch is that the data splits across two blobs, not one. ytInitialPlayerResponse is the authoritative source for raw machine-readable fields: integer view count, integer like count, ISO upload date, video duration in seconds. ytInitialData holds the formatted display strings that the page renders to the user: subscriber count ("478M subscribers"), view count with commas, the comment count display ("2.4M Comments").

We read both blobs and merge them into a single record. Comment count needs a special detour because it lives in a separate engagement panels array, not next to the other display fields.

Three constants for this script: the video URL, the output filename, and the same token we set earlier.

Reading ytInitialPlayerResponse and ytInitialData Together

Both blobs sit as consecutive var <name> = {...}; declarations in the watch page HTML. We use one helper to extract either:

Before the fetch we normalize any /shorts/<id> URL to /watch?v=<id>. The next H3 explains why; for now this is the four-line guard at the top of the script:

We make a single GET inline since this is the only HTTP call we need here:

The player response gives us videoDetails (videoId, title, author, channelId, lengthSeconds, viewCount, keywords, isLiveContent, isPrivate, thumbnails) and microformat.playerMicroformatRenderer (description, likeCount, uploadDate, publishDate, ownerProfileUrl, category, isFamilySafe, isShortsEligible).

The display strings come from ytInitialData. Three values we want, scattered across two renderer types:

Both forms of every value end up on the output object. The integer view count for analytics, the formatted display string for UI replication.

Handling /shorts/ URLs

We normalized the URL above without saying why. Here it is.

YouTube exposes the same Short under two URLs. /shorts/<id> serves the vertical full-screen overlay player. /watch?v=<id> serves the regular watch page with the same video.

The Shorts overlay uses a different layout where ytInitialData.contents is empty. The entire page state lives in an overlay-specific renderer that we are not parsing. ytInitialPlayerResponse is identical between the two URLs, but every renderer path we walk for display strings is missing on the Shorts version.

No /watch URL, no display strings.

After the rewrite at the top, every renderer path we already walked works without changes for both URL types. We will use the same trick again in the comments section.

Extracting Comment Count from the Engagement Panel

The formatted comment count display ("2.4M Comments") is not next to the other display strings in videoPrimaryInfoRenderer or videoSecondaryInfoRenderer. It lives in a separate engagementPanels array on ytInitialData, inside the panel whose panelIdentifier equals "engagement-panel-comments-section". The identifier is almost self-documenting, which is rare for YouTube internals.

The text comes back as either a runs array (more recent format) or a simpleText string (older format), so we handle both:

The integer comment count is not exposed anywhere on the watch page in any blob. Only the formatted display string. Getting the actual integer means either hitting the comments endpoint (we cover that next) or counting parsed comments after pagination.

Export to JSON

We assemble the output record as a single JSON object covering identity, title and description (raw and HTML), engagement counts (integer and display), timestamps (ISO and display), authorship, classification, and the thumbnail set. We keep both forms of every value, since different consumers want different things:

Then the timestamps, channel context, classification flags, and thumbnail set:

A run against the Rick Astley "Never Gonna Give You Up" watch URL gives us view_count: 1764213403, like_count: 18949050, comment_count_display: "2.4M Comments", 27 tags, category "Music", duration 213 seconds. The same code against a /shorts/<id> URL produces the same shape after the rewrite, with smaller numbers and tags: [] (Shorts rarely set keywords).

One page, two blobs, 25 fields. The watch page is now a single record.

Scraping YouTube Comments

Comments are the one surface that breaks the pattern. The watch page HTML does not contain any comment text. None. The page only ships the initial continuation token for the comments section. Trying to parse ytInitialData for comment data finds zero rows.

Here is the catch. The InnerTube response splits comment data across two parallel structures. A commentThreadRenderer array describes the thread layout and reply continuation tokens. A separate frameworkUpdates.entityBatchUpdate.mutations array stores the actual text and authorship inside commentEntityPayload objects. Same response, two arrays, joined by commentId. The architecture is smart for partial updates, a little painful to scrape.

Replies under each top-level comment require a second continuation request per thread. We do them inline as we walk the threads so the output structure ends up naturally nested.

Five constants for this script: the video URL, the comment and reply caps, the output filename, and the same token from before.

We add a small wrapper for the InnerTube body:

The body shape is identical for top-level pagination and for reply pagination. Only the token changes.

Finding the Initial Continuation Token

Same fetch shape as the video section, plus we grab the video ID for the output and the InnerTube client version for the POST body:

The watch page renderer list inside ytInitialData includes an itemSectionRenderer whose sectionIdentifier is "comment-item-section". Inside that section's contents is a continuationItemRenderer carrying the initial token for the comments feed.

We match by identifier instead of array position because the section order shifts when YouTube ships frontend changes:

If the section is missing because comments are turned off on the video, we raise SystemExit with a clear message. Failing loud beats failing quiet for the kind of bug that produces zero rows on a successful HTTP response.

Joining commentEntityPayload Mutations to Threads

Each InnerTube response from /youtubei/v1/next wraps items in either reloadContinuationItemsCommand (first page) or appendContinuationItemsAction (subsequent pages). We write one helper to handle both shapes:

What comes back is a list of commentThreadRenderer items (one per top-level comment) plus a continuationItemRenderer at the end (the next-page token). The thread items expose layout metadata, comment IDs, and reply tokens, but not the comment text or author.

That data lives in frameworkUpdates.entityBatchUpdate.mutations. Each mutation has a payload keyed by entity type. We only care about commentEntityPayload:

A commentEntityPayload packs everything we want per comment: text, published time, reply level, author display name and channel ID, avatar URL, verified/creator/artist flags, like count, reply count. We write a small parser to flatten those into a single dictionary:

The join is the easy part. For each thread item, read the commentId from inside its commentViewModel, look it up in the entity map, hand it to parse_comment. Both arrays arrive in the same response, so the lookup never misses on a healthy run.

Why Top-Level Threads and Reply Items Have Different Wrappers

A top-level thread item wraps commentViewModel twice. Yes, twice. The shape looks like this:

The actual data is in the inner commentViewModel. A reply item, returned by a reply continuation request, is exactly that inner piece standing alone: no commentThreadRenderer wrapper, no outer commentViewModel.

This means the lookup path differs between contexts. Top-level threads need it["commentThreadRenderer"]["commentViewModel"]["commentViewModel"]["commentId"]. Replies need it["commentViewModel"]["commentId"]. Mixing them up returns None and silently skips the comment with no error raised.

This is the single most error-prone part of the whole comments scraper. Each context needs its own code path:

Two snippets that look almost identical, except one has an extra .get("commentViewModel") at the top-level path. That one extra call is the difference between 0 comments and N comments parsed.

Fetching Reply Threads

Each commentThreadRenderer comes with a replies.commentRepliesRenderer.contents array. If that array contains a continuationItemRenderer, its token is the reply continuation for that thread. We post it to the same /youtubei/v1/next endpoint, which returns a flat list of commentViewModel items plus an optional next-page token for further reply pages:

We call fetch_replies once per top-level comment, immediately after parsing it, before moving to the next thread. That keeps the output structure cleanly nested but makes the request count grow fast. Forty top-level comments with up to five replies each on the Rick Astley video came out to 37 InnerTube POSTs total: two for the top-level pages (20 threads per page) and one per thread that has a reply continuation.

We tie everything together in the top-level pagination loop. For each item in the response we either parse a thread (and fetch its replies) or grab the next-page token:

The reply token sits inside the thread's replies block. We pull it, then call fetch_replies with it (or attach an empty array if the thread has no replies) and append the assembled comment:

The doubly-nested wrapper is on the third line of the outer loop. The flat wrapper is inside fetch_replies. Each path matches the shape of the surrounding context.

Export to JSON

We write the output as a single JSON file with the video ID, URL, a count, and an array of comment objects. Each comment holds a replies array of the same shape, with reply_level set to 1 instead of 0:

Forty top-level comments with up to five replies each on the Rick Astley video produces a 168-reply JSON tree under 40 threads (most threads return the full 5, a handful return fewer or none). The pinned @YouTube comment ("can confirm: he never gave us up") sits at position one with 226K likes and 962 reply count display.

What started as "comments do not even live in the page" ends as a JSON tree with replies nested under every thread.

Scraping YouTube Channel Pages

A channel page at /@<handle>/videos exposes the channel-level identity (ID, title, handle, description, vanity URL), display statistics (subscriber count, total video count), media URLs (avatar, banner), and the first 30 uploaded videos. Subsequent video pages come from InnerTube /youtubei/v1/browse with continuation tokens, the same recipe as search and comments with a different endpoint path.

Four constants for this script: the channel URL, the video cap, the output filename, and the same token from before.

Channel URLs come in different shapes. @<handle>, /channel/<id>, with or without the trailing /videos. We auto-append /videos if it is missing so the response always uses the channel-page layout with the Videos tab pre-selected:

The channel-level data and the video list live in different parts of ytInitialData. We read them in two passes and merge them into one output object. The same fetch helper from earlier handles both the initial GET and every continuation POST.

Channel Header vs. Videos Tab

Same fetch shape as the comments section, then we extract the InnerTube client version (for the continuation POSTs) and ytInitialData (which carries both the header and the first page of videos):

Channel-level metadata splits across two places. metadata.channelMetadataRenderer ships the stable identity fields (external ID, title, description, channel URL, vanity URL, keywords, family-safe flag). A deeply nested chain under header.pageHeaderRenderer stores the display-string statistics (handle, subscriber count, total videos).

The display rows are positional. Row zero, part zero is the @handle. Row one, part zero is the subscriber count display. Row one, part one is the total videos display. Channels with very low engagement may have shorter rows, so we check array lengths before indexing:

Then the avatar and banner pulls (the next H3 explains why those paths run so deep), followed by the dict assembly:

We call it on the extracted blob:

The video list lives under a tab structure. twoColumnBrowseResultsRenderer.tabs is an array of six tabs (Home, Videos, Shorts, Playlists, Posts, Search). We pick the tab whose title equals "Videos" instead of relying on a position index, because the order shifts when channels disable certain tabs or YouTube reorders them:

Each video on the tab is a richItemRenderer containing a videoRenderer with the same field shape we already parsed for search results. We reuse that shape, dropping the channel name since channel context is implicit on a channel page:

Pulling Avatar and Banner URLs from pageHeaderViewModel

The avatar (channel profile picture) is buried six levels deep inside pageHeaderViewModel. Six levels deep for one image URL is aggressive even by YouTube standards. The sources array at the bottom carries 2 to 3 size variants (typically 72×72, 120×120, 160×160), and we want the largest.

The banner (channel header image) is four levels deep, under pageHeaderViewModel.banner, with 5 to 7 size variants from 1060×175 (mobile) up to 2560×424 (desktop ultrawide).

Both URLs point at yt3.googleusercontent.com with rendering directives baked into the path (=s120-c-k-c0x00ffffff-no-rj for avatars, =w2560-fcrop64=1,... for banners). The [-1] index in the dict above picks the largest variant in each list. The conditional handles the rare case of a channel with no banner uploaded: when banner_sources is empty, the banner field comes out as None.

Paginating the Videos Tab

The Videos tab opens with 30 richItemRenderer items followed by one continuationItemRenderer. The token from that last item posts to InnerTube /youtubei/v1/browse with the same body shape we used for search and comments, just a different endpoint path.

We write a small helper to walk any item array (initial or continuation) and return the next token if present:

The InnerTube /youtubei/v1/browse response uses both onResponseReceivedActions (newer YouTube responses) and onResponseReceivedCommands (older format). We concatenate both before iterating, which handles either shape without branching:

The loop stops when the row count hits the cap or when a response returns no items. MAX_VIDEOS = 60 produces two pagination passes (30 + 30) for any active channel.

Export to JSON

We bundle the channel info object, an integer count, and the video array into one JSON file:

A run against https://www.youtube.com/@MrBeast with MAX_VIDEOS = 60 writes a JSON file with the channel header (478M subscribers, 966 total videos, the avatar and banner URLs) plus 60 videos from two pagination passes. The same code runs unchanged against smaller channels: @Fireship returns 4.18M subscribers and 803 videos, @leerob returns 78K subscribers and 109 videos.

Channel header on the outside, paginated video list on the inside. The channel page is now structured.

Conclusion

YouTube ships data through three mechanisms across the four surfaces we covered. ytInitialData holds the rendered page state on every page type. ytInitialPlayerResponse handles the raw machine-readable fields on watch pages. The InnerTube endpoints (/youtubei/v1/search, /youtubei/v1/browse, /youtubei/v1/next) serve every paginated continuation through POST requests with continuation tokens.

The same two-line regex pulls embedded blobs out of every page type. One WEB client version powers every InnerTube call across all three endpoints. One proxy parameter (super=true) covers the anti-bot layer for everything else. From here we scale the boring way: swap a single search query, video URL, or channel URL for a list, throttle the request rate, and watch the InnerTube client version because YouTube rotates it weekly.

FAQ

Does Scrape.do have a managed YouTube API?

Yes. The YouTube API (/plugin/google/youtube) is a managed endpoint that returns the full search page as structured JSON in one HTTP call: videos, channels, playlists, Shorts shelves, ads, and discovery rails. Views come back as parsed integers, extensions: ["4K"] is already extracted, and the sp parameter handles every filter (sort, result type, duration, features, upload date) plus pagination cursors. Each call costs 10 credits flat. Use the manual approach in this article when you want to learn the InnerTube model or pull fields the plugin doesn't expose; use the plugin in production.

Does YouTube block web scraping?

Yes. YouTube fingerprints every request against IP reputation, browser fingerprint, TLS handshake, and JavaScript challenge solving. A clean Python session lands on a cookie consent wall (HTTP 200 with no ytInitialData) instead of the real page. Routing requests through Scrape.do with super=true bypasses the wall.

What is InnerTube and why does it matter for scraping?

InnerTube is YouTube's internal JSON API used by the website to load any data after the first page render. It powers search pagination, channel video lists, comment loading, and Shorts feeds. It matters because the first page of any list loads in HTML, but every subsequent page requires an InnerTube POST. A scraper that only parses the embedded JSON gets the first 18 to 30 items and nothing else.

Why do comments load through a separate API instead of the page HTML?

YouTube's comments architecture uses a streaming-update pattern. The watch page only embeds the initial continuation token for the comments section; the actual comment payloads stream in through /youtubei/v1/next POSTs as the user scrolls. A commentThreadRenderer array describes thread layout and reply continuations, while a parallel frameworkUpdates.entityBatchUpdate.mutations array stores the actual text and author. Both arrive in the same response but require a join by commentId.

How do I scrape YouTube Shorts data?

Shorts appear in two contexts. As items inside regular search results, they live in a gridShelfViewModel shelf using the shortsLockupViewModel schema, which exposes only video ID, title, view count, and thumbnail. As standalone pages at /shorts/<id>, they serve a different HTML layout where ytInitialData.contents is empty. The fix is to rewrite any /shorts/<id> URL to /watch?v=<id> before fetching, which returns the full watch-page layout.

Why is trending not covered?

YouTube deprecated the unauthenticated /feed/trending page in 2024 and announced in 2025 it is retiring the Trending Tab entirely. The remaining /category/trending URLs (gaming, music, news, etc.) are editorial catalog feeds, not real-time popularity data: they surface evergreen content alongside recent uploads. For real-time popularity, the recommended approach is a search query with sp=CAMSAhAB (the URL-encoded filter for "this week, sorted by view count") which the search code supports through the SEARCH_QUERY constant. YouTube Music charts cover music-only popularity.