diff --git a/Extractors.md b/Extractors.md index 91e5f3c..e2f97a7 100644 --- a/Extractors.md +++ b/Extractors.md @@ -1,54 +1,33 @@ # YouTube +> [!IMPORTANT] +> YouTube is gradually enforcing the use of a "PO Token" to be able to download videos. Due to the nature of these tokens, yt-dlp cannot generate them and they must be provided externally. +> +> By default, yt-dlp will attempt to download videos using clients that do not currently require a PO Token. However, some formats and features may not be available without the token(s). +> +> At this time, it is **recommended** to use provide a PO Token to use with the `web` client. Refer to the [PO Token Guide](https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide) on how to set up yt-dlp for this. ## Exporting YouTube cookies +> [!CAUTION] +> By using your account with yt-dlp, you run the risk of it being banned (temporarily or permanently). +> Be mindful with the request rate and amount of downloads you make with an account. Use it only when necessary, or consider using a throwaway account. + +> [!NOTE] +> This is only necessary for content that requires an account to access, such as private playlists, age-restricted videos and members-only content. + If you are unfamiliar with the basics of exporting cookies and passing them to yt-dlp, then first see [How do I pass cookies to yt-dlp?](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp) -YouTube rotates cookies frequently on open YouTube browser tabs as a security measure. +YouTube rotates account cookies frequently on open YouTube browser tabs as a security measure. To export cookies that will remain working with yt-dlp, you will need to export cookies in such a way that they are never rotated. One way to do this is through a private browsing/incognito window: 1. Open a new private browsing/incognito window and log into YouTube 2. Open a new tab and **close the YouTube tab** -3. Export cookies from the browser then **close the private browsing/incognito window** so the session is never opened in the browser again. - -## Logging in with OAuth - -> [!CAUTION] -> Due to new restrictions enacted by YouTube, logging in with OAuth no longer works with yt-dlp. [You should use cookies instead.](#exporting-youtube-cookies) +3. Export `youtube.com` cookies from the browser then **close the private browsing/incognito window** so the session is never opened in the browser again. -## PO Token Guide - -Proof of Origin (PO) token is a parameter that YouTube requires to be sent with video playback requests from some clients. Without it, requests for the affected clients' format URLs may return HTTP Error 403, or result in your account or IP address being blocked. - -This token is generated by BotGuard (Web) / DroidGuard (Android) to attest the requests are coming from a genuine client. This guide will be assuming the use of a PO Token generated by BotGuard, for use with the `web` client. - -The token is only valid for a limited time (usually at least 12 hours), so it will need to be refreshed periodically. - -> For PO Tokens generated by BotGuard, when logged out, the PO Token is bound to a Visitor ID. This Visitor ID is found in the `VISITOR_INFO1_LIVE` cookie, or in the `visitorData` value which is sent with Innertube API requests. When logged in, the PO Token is bound to the account Session ID (part of the Data Sync ID), rather than the Visitor ID. - - -### Manually acquiring a PO Token from a browser for use when logged out - -This process goes through manually obtaining a PO Token generated on YouTube in a web browser, and then manually passing that to yt-dlp via the `po_token` extractor argument. - -Steps: - -1. Open [YouTube Music](https://music.youtube.com) or YouTube Embedded (e.g. https://www.youtube.com/embed/aqz-KE-bpKQ) in a browser. **Make sure you are not logged in to any account!** -2. Open any video -3. Open the developer console (F12), then go to the "Network" tab and filter by `v1/player` -4. Click the video to play and a player request will appear in the network tab -5. In the request payload JSON, find the PO Token at `serviceIntegrityDimensions.poToken` and save that value -6. Export cookies from the browser -7. Pass the PO Token to yt-dlp using `--extractor-args "youtube:player-client=web,default;po_token=web+PO_TOKEN_VALUE_HERE"` with cookies (`--cookies COOKIES_FILE`) - -Addendum: -- You can also get the PO Token from any of the `videoplayback` URLs (it is the `pot` query parameter). -- You can also use `--cookies-from-browser` if that is pointed to the browser session with the same logged-out session you extracted the PO Token from. - -#### Passing Visitor Data without cookies +## Passing Visitor Data without cookies In some cases, you may not want to use cookies and instead pass Visitor Data to use in Innertube API requests. @@ -57,30 +36,14 @@ In some cases, you may not want to use cookies and instead pass Visitor Data to You can do this with: - --extractor-args "youtube:player-client=web,default;player-skip=webpage,configs;po_token=web+PO_TOKEN_VALUE_HERE;visitor_data=VISITOR_DATA_VALUE_HERE" + --extractor-args "youtubetab:skip=webpage" --extractor-args "youtube:player_skip=webpage,configs;visitor_data=VISITOR_DATA_VALUE_HERE" -### Manually acquiring a PO Token from a browser for use when logged in +## PO Token Guide -The process for obtaining a PO Token for use when yt-dlp is logged into an account is similar to the above. The PO Token obtained should work with either [cookies](#exporting-youtube-cookies) or [OAuth](#logging-in-with-oauth). +[This section has been moved to a dedicated page](https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide) -Steps: +## Logging in with OAuth -1. Open [YouTube Music](https://music.youtube.com) in a browser, and log in with the user you are using with yt-dlp -2. Open any video -3. Follow steps 3-5 [above](#manually-acquiring-a-po-token-from-a-browser-for-use-when-logged-out) -4. Pass the PO Token to yt-dlp using `--extractor-args "youtube:player-client=web;po_token=web+PO_TOKEN_VALUE_HERE"` with your method of auth (cookies or OAuth) - -Note only the `web` client is used above example (instead of `web` and `default`). When logged in, it is **recommended** to explicitly supply a PO token to all clients used to help avoid getting your account blocked. -- If you want to use another client such as `mweb`, you should append `mweb+PO_TOKEN_VALUE_HERE` to the `po_token` argument and append `mweb` to the `player-client` argument. - -If you are using cookies, see [Exporting YouTube Cookies](#exporting-youtube-cookies) on how to export cookies without them being invalidated. - - -### Other methods of acquiring a PO Token - -#### Plugins -- [yt-dlp-get-pot](https://github.com/coletdjnz/yt-dlp-get-pot) by [coletdjnz](https://github.com/coletdjnz) - - A plugin framework for yt-dlp to support fetching PO Tokens from external providers -- [bgutil-ytdlp-pot-provider](https://github.com/Brainicism/bgutil-ytdlp-pot-provider) by [Brainicism](https://github.com/Brainicism) - - A [yt-dlp-get-pot](https://github.com/coletdjnz/yt-dlp-get-pot) provider which uses [BgUtils](https://github.com/LuanRT/BgUtils) to generate PO Tokens \ No newline at end of file +> [!CAUTION] +> Due to new restrictions enacted by YouTube, logging in with OAuth no longer works with yt-dlp. [You should use cookies instead.](#exporting-youtube-cookies) diff --git a/PO Token Guide.md b/PO Token Guide.md new file mode 100644 index 0000000..a642682 --- /dev/null +++ b/PO Token Guide.md @@ -0,0 +1,164 @@ +[Go to YouTube Extractor Wiki](https://github.com/yt-dlp/yt-dlp/wiki/Extractors#youtube) + +# YouTube PO Token Guide + +> [!TIP] +> TL;DR recommended setup: Either use a [plugin](#plugins), or [manual extraction](#guide-providing-a-po-token-manually-for-use-with-web-client), to provide the `web` client with a [PO Token for GVS requests](#cases-where-a-po-token-is-required). +> +> See the notice in the [YouTube Extractor Wiki](https://github.com/yt-dlp/yt-dlp/wiki/Extractors#youtube) for more information on the current state of downloading videos from YouTube. + + + +* [YouTube PO Token Guide](#youtube-po-token-guide) + * [Introduction](#introduction) + * [Cases where a PO Token is required](#cases-where-a-po-token-is-required) + * [Current PO Token enforcement](#current-po-token-enforcement) + * [Guide: Providing a PO Token manually (for use with `web` client)](#guide-providing-a-po-token-manually-for-use-with-web-client) + * [PO Token for GVS](#po-token-for-gvs) + * [No account](#no-account) + * [With an account](#with-an-account) + * [PO Token for Player](#po-token-for-player) + * [Plugins](#plugins) + * [Other PO Token related tools](#other-po-token-related-tools) + * [Technical details](#technical-details) + * [PO Tokens for GVS](#po-tokens-for-gvs) + * [PO Tokens for Player](#po-tokens-for-player) + + + +## Introduction + +Proof of Origin (PO) Token is a parameter that YouTube requires to be sent with requests from some clients. Without it, requests for the affected clients' format URLs may return HTTP Error 403, or result in your account or IP address being blocked. + +A PO Token is generated by an attestation provider on Web, Android and iOS platforms to attest the requests are coming from a genuine client. + +For yt-dlp, you can provide PO Token(s) using the [`po_token` extractor argument](https://github.com/yt-dlp/yt-dlp#youtube), or you can [install a plugin](#plugins) that hooks into the YouTube extractor to handle fetching a PO Token. + +For more technical details on these tokens, refer to the [technical details section](#technical-details). + +## Cases where a PO Token is required + +There are currently two cases yt-dlp may require PO Tokens for video downloads, [depending on the client used](#current-po-token-enforcement): +- GVS: Google Video Server requests +- Player: Innertube `player` requests + +The PO Tokens may be generated differently for each of these cases, depending on the client. + +### Current PO Token enforcement + +YouTube is at present rolling out changes to enforce PO Tokens for video playback. Currently, only GVS requires PO Tokens for some clients. + +| Client | PO Token for GVS Required | PO Token for Player Required | Notes | +|----------------|---------------------------|------------------------------|---------------------------------------| +| `web` | Yes | No | | +| `web_safari` | Yes* | No | | +| `mweb` | Yes | No | | +| `tv` | No | No | | +| `tv_embedded` | No | No | Requires account cookies | +| `web_embedded` | No | No | | +| `web_music` | Yes | No | | +| `web_creator` | Yes | No | Requires account cookies | +| `android` | Yes | ? | Account cookies not supported | +| `android_vr` | No | No | YouTube Kids videos are not available | +| `ios` | Yes* | ? | Account cookies not supported | + +*Client provides HLS (m3u8) formats which do not require PO Token for GVS at this time. + +You can select what client to use with the [`player_client` extractor argument](https://github.com/yt-dlp/yt-dlp#youtube). + +## Guide: Providing a PO Token manually (for use with `web` client) + +This section provides a basic guide on extracting PO Token(s) manually from YouTube in a web browser **for use with the `web` client**, and manually passing it to yt-dlp via the [`po_token` extractor argument](https://github.com/yt-dlp/yt-dlp#youtube). +The same PO Token extraction method _may_ work with other web browser-based clients too. + +> [!TIP] +> When supplying multiple PO Tokens, use the same extractor args option and comma-separate the PO Token configurations. For example: +> `--extractor-args "youtube:po_token=web.gvs+GVS_PO_TOKEN_VALUE_HERE,web.player+PLAYER_PO_TOKEN_VALUE_HERE"` + + +### PO Token for GVS + +The PO Token used for `web` GVS requests is tied to your YouTube session. It generated differently depending on if you are logged in to yt-dlp or not. + +#### No account + +1. Open [YouTube Music](https://music.youtube.com) or YouTube Embedded (e.g. https://www.youtube.com/embed/aqz-KE-bpKQ) in a browser. **Make sure you are not logged in to any account!** +2. Open any video +3. Open the developer console (F12), then go to the "Network" tab (click the `>>` button if you don't see it) +4. Filter requests by `googlevideo.com` +5. Click the video and play for a few seconds - requests to `googlevideo.com` should appear in the network tab +6. From the most recent `googlevideo.com` request, extract the `pot` query parameter value from the URL +7. Pass the PO Token for GVS to yt-dlp using `--extractor-args "youtube:po_token=web.gvs+PO_TOKEN_VALUE_HERE"` with cookies (`--cookies COOKIES_FILE` or `--cookies-from-browser`) + +Although not recommended, you may also provide visitor data instead of cookies. Refer to [Passing Visitor Data without cookies](https://github.com/yt-dlp/yt-dlp/wiki/Extractors#passing-visitor-data-without-cookies). + +#### With an account + +1. Open [YouTube Music](https://music.youtube.com) in a browser, and log in with the user you are using with yt-dlp +2. Open any video +3. Follow steps 3-6 above +4. Pass the PO Token for GVS to yt-dlp using `--extractor-args "youtube:po_token=web.gvs+PO_TOKEN_VALUE_HERE"` [with your account cookies ](https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies) + +Addendum: +- If there is no `pot` parameter in the `googlevideo.com` URL, wait a few seconds for more requests to be made and check them. +- If there is a `sabr=1` query parameter in the `googlevideo.com` URL, then the PO Token is in the request body protobuf (and therefore is not easy to extract). + - This should not be the case for YouTube Music or YouTube Embedded - if so, please raise an issue, so we can update these docs. + + +### PO Token for Player + +The PO Token for `web` Player requests is tied to the Video ID. This means you must generate a new PO Token for each video. + +> [!NOTE] +> If you are using the `web` client and have not disabled the `webpage` request, providing this PO Token is not necessary at this time. + +1. Open [YouTube Web](https://www.youtube.com) in a browser. +2. Open the developer console (F12), then go to the "Network" tab and filter by `v1/player` +3. Navigate to the video you want to download (e.g. using search - do not go to the video url directly as the page will refresh) +4. In the request payload JSON, find the PO Token at `serviceIntegrityDimensions.poToken` and save that value +5. Export cookies from the browser +6. Pass the PO Token for Player to yt-dlp using `--extractor-args "youtube:po_token=web.player+PO_TOKEN_VALUE_HERE"` + +## Plugins + +Manually fetching PO Tokens can be a tedious process. Alternatively, you can use a plugin that hooks into the YouTube extractor to handle fetching a PO Token. + +- [yt-dlp-get-pot](https://github.com/coletdjnz/yt-dlp-get-pot) by [coletdjnz](https://github.com/coletdjnz) + - An experimental plugin framework for yt-dlp to support fetching PO Tokens from external providers. _Maintained by a yt-dlp maintainer._ +- [bgutil-ytdlp-pot-provider](https://github.com/Brainicism/bgutil-ytdlp-pot-provider) by [Brainicism](https://github.com/Brainicism) + - A [yt-dlp-get-pot](https://github.com/coletdjnz/yt-dlp-get-pot) provider which uses [BgUtils](https://github.com/LuanRT/BgUtils) to generate PO Tokens. _Not affiliated with yt-dlp._ + +See the [yt-dlp-plugins GitHub topic](https://github.com/topics/yt-dlp-plugins) and the [yt-dlp plugin wiki](https://github.com/yt-dlp/yt-dlp/wiki/Plugins) to explore more yt-dlp plugins. + +## Other PO Token related tools + +- [BgUtils](https://github.com/LuanRT/BgUtils) by [LuanRT](https://github.com/LuanRT) + - Utility to generate PoTokens and run BotGuard attestation challenges. +- [YouTube Trusted Session Generator](https://github.com/iv-org/youtube-trusted-session-generator) by [Invidious](https://github.com/iv-org) + - A generator for getting a session that passes all the checks from YouTube side + +## Technical details + +PO Tokens are still an active area of research, and at the same time YouTube is continuously updating the implementation and enforcement. The following information is based on the current understanding of PO Tokens, and is subject to change. + +A PO Token is generated by either BotGuard (Web), DroidGuard (Android), iOSGuard (iOS). A PO Token from one platform cannot be used on another (i.e., Web PO Token cannot be used on Android or iOS). + +If you are interested, we recommend checking out the [BgUtils](https://github.com/LuanRT/BgUtils) project which does a deeper dive into the BotGuard attestation process. + +### PO Tokens for GVS + +GVS uses a PO token bound to the user session. + +For Web session tokens: +- When logged out, are bound to a Visitor ID. This Visitor ID is found in the `VISITOR_INFO1_LIVE` cookie, in the `visitorData` value which is sent with Innertube API requests, or with `ytcfg.get('VISITOR_DATA')` in the browser console. +- When logged in, are bound to the account Session ID (first part of the Data Sync ID). This can be found at `responseContext.mainAppWebResponseContext.dataSyncId` in Innertube responses, or with `ytcfg.get('DATASYNC_ID')` in the browser console. + +If a video download fails with an HTTP 403 midway through, then the client likely requires a PO Token for GVS. + +These PO Tokens are only valid for a limited time (usually at least 12 hours), so it will need to be refreshed periodically. However, some reports suggest that the token may be valid for many days. + +### PO Tokens for Player + +Web Tokens for player requests are bound to the video ID the associated `/player` request is for. YouTube has only recently started generating these for some clients (e.g. `web`). + +As of writing, some clients, such as `web_music` use a session-bound PO Token for player requests. This is likely to change.