Commit graph

1330 commits

Author SHA1 Message Date
coletdjnz 69d3191495
[test] Skip source address tests if the address cannot be bound to (#8900)
Fixes https://github.com/yt-dlp/yt-dlp/issues/8890

Authored by: coletdjnz
2024-01-20 10:39:49 +13:00
Kenshin bcec568ea7 Add ffmpeg progress tracking to FFmpegFD
Add ffmpeg progress tracking to FFmpegPostProcessor

Apply changes from the code review

Fix a bug where the subprocess didn't capture any output thus an empty stdout and stderr were sent back

Add missing hooks

Revert "Add missing hooks"

This reverts commit a359c5ea10bb35b965e80801e736f43cdbcf3294.

Add support of -ss=132 timestamp format

Infer filename from ffmpeg args instead of info_dic

Remove redundant parenthesis and switch from to_stodout to to_screen

Add info kwarg with multiple files and ffmpeg to track progress

Moved format progress function to util

Moved format progress function to util

Add progress tracking to postprocessing operations

Fix typing error

Handle self._downloader is None at __init__

Move format progress functions to utils

Move format progress functions to utils

Handle case where ydl passed is None

Handle case where ydl passed is None

Handle case where _multiline isn't initialized

Handle case where _multiline isn't initialized

Fix streams incorrectly returned

Fix case where ydl is nested in the downloader

Add progress_hook attribute

Fix bug after merge

Fix import bugs after merge

Catch up with upstream

Fix merge errors #1

Adapt tests and implementatation for ffmpeg progress tracking args
2024-01-17 15:14:41 +01:00
Simon Sawicki ffbd4f2a02
[utils] traverse_obj: Support xml.etree.ElementTree.Element (#8911)
Authored by: Grub4K
2024-01-05 21:26:17 +01:00
Simon Sawicki f9fb3ce86e
[cleanup] Misc (#8598)
Authored by: bashonly, pukkandan, seproDev, Grub4K

Co-authored-by: bashonly <bashonly@protonmail.com>
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
Co-authored-by: sepro <4618135+seproDev@users.noreply.github.com>
2023-12-30 22:27:36 +01:00
Simon Sawicki 00cdda4f6f
[core] Fix format selection parse error for CPython 3.12 (#8797)
Authored by: Grub4K
2023-12-24 22:09:01 +01:00
bashonly 632b8ee54e
[core] Release workflow and Updater cleanup (#8640)
- Only use trusted publishing with PyPI and remove support for PyPI tokens from release workflow
- Clean up improper actions syntax in the build workflow inputs
- Refactor Updater to allow for consistent unit testing with `UPDATE_SOURCES`

Authored by: bashonly
2023-12-21 21:06:26 +00:00
bashonly 37755a037e
[test:networking] Update tests for OpenSSL 3.2 (#8814)
Authored by: bashonly
2023-12-20 19:03:54 +00:00
coletdjnz 196eb0fe77
[networking] Strip whitespace around header values (#8802)
Fixes https://github.com/yt-dlp/yt-dlp/issues/8729
Authored by: coletdjnz
2023-12-20 19:15:38 +13:00
Simon Sawicki 0b6f829b1d
[utils] traverse_obj: Move is_user_input into output template (#8673)
Authored by: Grub4K
2023-12-06 21:46:45 +01:00
pukkandan 993edd3f6e
[outtmpl] Support multiplication
Related: #8683
2023-12-06 03:44:11 +05:30
sepro 1732eccc0a
[core] Parse release_year from release_date (#8524)
Closes #7263
Authored by: seproDev
2023-11-26 02:12:05 +00:00
coletdjnz ccfd70f4c2
[rh:websockets] Migrate websockets to networking framework (#7720)
* Adds a basic WebSocket framework
* Introduces new minimum `websockets` version of 12.0
* Deprecates `WebSocketsWrapper`

Fixes https://github.com/yt-dlp/yt-dlp/issues/8439

Authored by: coletdjnz
2023-11-20 08:04:04 +00:00
sepro 45d82be65f
[ie/nebula] Overhaul extractors (#8566)
Closes #4300, Closes #5814, Closes #7588, Closes #6334, Closes #6538
Authored by: elyse0, pukkandan, seproDev

Co-authored-by: Elyse <26639800+elyse0@users.noreply.github.com>
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
2023-11-20 01:03:33 +00:00
bashonly f4b95acafc
Remove Python 3.7 support (#8361)
Closes #7803
Authored by: bashonly
2023-11-16 18:39:00 +00:00
bashonly f04b5bedad
[ie] Do not smuggle http_headers
See: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-3ch3-jhc6-5r8x

Authored by: coletdjnz
2023-11-14 22:04:25 +01:00
bashonly 87264d4fda
[test:update] Implement simple updater unit tests
Authored by: bashonly
2023-11-12 18:30:55 -06:00
Simon Sawicki 4e38e2ae9d
[rh:requests] Handle both bytes and int for IncompleteRead.partial (Fix 8a8b54523a) (#8348)
Authored by: bashonly, coletdjnz, Grub4K
2023-10-15 10:54:38 +02:00
coletdjnz 8a8b54523a
[rh:requests] Add handler for requests HTTP library (#3668)
Adds support for HTTPS proxies and persistent connections (keep-alive)

Closes https://github.com/yt-dlp/yt-dlp/issues/1890
Resolves https://github.com/yt-dlp/yt-dlp/issues/4070
Resolves https://github.com/ytdl-org/youtube-dl/issues/32549
Resolves https://github.com/ytdl-org/youtube-dl/issues/14523
Resolves https://github.com/ytdl-org/youtube-dl/issues/13734

Authored by: coletdjnz, Grub4K, bashonly
2023-10-13 23:33:00 +00:00
Awal Garg 9d7ded6419
[utils] js_to_json: Fix Date constructor parsing (#8295)
Authored by: awalgarg, Grub4K
2023-10-08 01:57:23 +02:00
Simon Sawicki 088add9567
[cleanup] Misc
Authored by: Grub4K
2023-09-24 02:35:23 +02:00
Simon Sawicki de015e9307
[core] Prevent RCE when using --exec with %q (CVE-2023-40581)
The shell escape function is now using `""` instead of `\"`. `utils.Popen` has been patched to properly quote commands.

Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg for reference.

Authored by: Grub4K
2023-09-24 02:29:01 +02:00
Simon Sawicki 61bdf15fc7
[core] Raise minimum recommended Python version to 3.8 (#8183)
Authored by: Grub4K
2023-09-24 02:24:47 +02:00
bashonly 5ca095cbcd
[cleanup] Misc (#8182)
Closes #7796, Closes #8028
Authored by: barsnick, sqrtNOT, gamer191, coletdjnz, Grub4K, bashonly
2023-09-23 20:00:31 +00:00
bashonly 661c9a1d02
[test:download] Test for expected_exception
Authored by: at-wat

Co-authored-by: Atsushi Watanabe <atsushi.w@ieee.org>
2023-09-21 17:48:57 -05:00
bashonly 52414d64ca
[utils] js_to_json: Handle Array objects
Authored by: Grub4K, std-move

Co-authored-by: std-move <26625259+std-move@users.noreply.github.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
2023-09-21 16:51:57 -05:00
coletdjnz 20fbbd9249
[networking] Fix various socks proxy bugs (#8065)
- Fixed support for IPv6 socks proxies
- Fixed support for IPv6 over socks5
- Fixed --source-address not being obeyed for socks4 and socks5
- Fixed socks4a when the destination address is an IPv4 address

Closes https://github.com/yt-dlp/yt-dlp/issues/7959
Fixes https://github.com/ytdl-org/youtube-dl/issues/15368

Authored by: coletdjnz
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
Co-authored-by: bashonly <bashonly@bashonly.com>
2023-09-18 07:33:26 +00:00
coletdjnz fcd6a76adc
[tests] Add tests for socks proxies (#7908)
Authored by: coletdjnz
2023-08-25 07:10:44 +00:00
coletdjnz db7b054a61
[networking] Add request handler preference framework (#7603)
Preference functions that take a request and a request handler instance can be registered to prioritize different request handlers per request.

Authored by: coletdjnz
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
2023-08-04 22:17:48 +00:00
pukkandan 6148833f5c
[cleanup] Misc 2023-07-30 04:06:18 +05:30
pukkandan 3f7965105d
[utils] HTTPHeaderDict: Handle byte values 2023-07-30 03:18:10 +05:30
pukkandan de20687ee6
[test] Fix test_load_certifi
Closes #7688, #7675
2023-07-29 21:53:00 +05:30
coletdjnz 4bf912282a
[networking] Remove dot segments during URL normalization (#7662)
This implements RFC3986 5.2.4 remove_dot_segments during the URL normalization process.

Closes #3355, #6526

Authored by: coletdjnz
2023-07-28 22:40:20 +00:00
coletdjnz bbeacff7fc
[networking] Ignore invalid proxies in env (#7704)
Authored by: coletdjnz
2023-07-27 20:26:02 +05:30
coletdjnz 95abea9a03
[test] Fix httplib_validation_errors test for old Python versions (#7677)
Fixes https://github.com/yt-dlp/yt-dlp/issues/7674

Authored by: coletdjnz
2023-07-24 19:18:52 +00:00
coletdjnz 86aea0d3a2
[networking] Add strict Request extension checking (#7604)
Authored by: coletdjnz
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
2023-07-23 05:17:15 +00:00
pukkandan a250b24733
[compat] Ensure submodules are imported correctly
Closes #7663
2023-07-22 18:10:35 +05:30
pukkandan 62b5c94cad
[cleanup] Misc fixes
Closes #7528
2023-07-22 09:09:52 +05:30
pukkandan a264433c9f
[outtmpl] Fix replacement for playlist_index 2023-07-22 08:57:43 +05:30
bashonly e57eb98222
[fd/external] Fix ffmpeg input from stdin (#7655)
Bugfix for 1ceb657bdd

Authored by: bashonly
2023-07-22 02:32:49 +00:00
bashonly af86873218
[utils] Improve parse_duration
Authored by: bashonly
2023-07-20 08:40:31 -05:00
bashonly 75dc8e673b
[networking] Fix --legacy-server-connect (#7645)
Bugfix for 227bf1a33b

Authored by: bashonly
2023-07-20 13:31:17 +00:00
bashonly 71baa490eb
[networking] Fix POST requests with zero-length payloads (#7648)
Bugfix for 227bf1a33b

Authored by: bashonly
2023-07-20 13:23:30 +00:00
bashonly 42ded0a429
[fd/external] Fixes to cookie handling
- Fix bug in `axel` Cookie header arg
- Pass cookies to `curl` as strings
- Write session cookies for `aria2c` and `wget`

Closes #7539
Authored by: bashonly
2023-07-15 15:25:51 -05:00
bashonly 6c5211cebe
[core] Fix HTTP headers and cookie handling
- Remove `Cookie` header from `http_headers` immediately after loading into cookiejar
- Restore compat for `--load-info-json` cookies
- Add more tests
- Fix improper passing of Cookie header by `MailRu` extractor

Closes #7558
Authored by: bashonly, pukkandan
2023-07-15 15:25:45 -05:00
coletdjnz 3d2623a898
[compat, networking] Deprecate old functions (#2861)
Authored by: coletdjnz, pukkandan
2023-07-15 16:18:35 +05:30
coletdjnz 227bf1a33b
[networking] Rewrite architecture (#2861)
New networking interface consists of a `RequestDirector` that directs
each `Request` to appropriate `RequestHandler` and returns the
`Response` or raises `RequestError`. The handlers define adapters to
transform its internal Request/Response/Errors to our interfaces.

User-facing changes:
- Fix issues with per request proxies on redirects for urllib
- Support for `ALL_PROXY` environment variable for proxy setting
- Support for `socks5h` proxy
   - Closes https://github.com/yt-dlp/yt-dlp/issues/6325, https://github.com/ytdl-org/youtube-dl/issues/22618, https://github.com/ytdl-org/youtube-dl/pull/28093
- Raise error when using `https` proxy instead of silently converting it to `http`

Authored by: coletdjnz
2023-07-15 16:18:35 +05:30
pukkandan c365dba843
[networking] Add module (#2861)
No actual changes - code is only moved around
2023-07-15 16:18:34 +05:30
pukkandan 1b392f905d
[utils] Add temporary shim for logging
Related: #5680, #7517
2023-07-15 16:18:34 +05:30
Mahmoud Abdel-Fattah 2af4eeb772
[utils] clean_podcast_url: Handle more trackers (#7556)
Authored by: mabdelfattah, bashonly
Closes #7544
2023-07-11 06:30:38 +05:30
Simon Sawicki 3121512228
[core] Change how Cookie headers are handled
Cookies are now saved and loaded under `cookies` key in the info dict
instead of `http_headers.Cookie`. Cookies passed in headers are
auto-scoped to the input URLs with a warning.

Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj

Authored by: Grub4K
2023-07-06 23:14:39 +05:30
coletdjnz f8b4bcc0a7
[core] Prevent Cookie leaks on HTTP redirect
Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj

Authored by: coletdjnz
2023-07-06 23:14:39 +05:30
bashonly 1ceb657bdd
[fd/external] Scope cookies
- ffmpeg: Calculate cookies from cookiejar and pass with `-cookies` arg instead of `-headers`
- aria2c, curl, wget: Write cookiejar to file and use external FD built-in cookiejar support
- httpie: Calculate cookies from cookiejar instead of `http_headers`
- axel: Calculate cookies from cookiejar and disable http redirection if cookies are passed
    - May break redirects, but axel simply don't have proper cookie support

Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj

Authored by: bashonly, coletdjnz
2023-07-06 23:14:38 +05:30
pukkandan 47bcd43724
[outtmpl] Pad playlist_index etc even when with internal formatting
Closes #7501
2023-07-06 20:22:03 +05:30
pukkandan cd810afe2a
[extractor/youtube] Improve nsig function name extraction 2023-06-22 13:27:18 +05:30
pukkandan 42f2d40b47
Update to ytdl-commit-07af47
[YouTube] Improve fix for ae8ba2c
07af47960f
2023-06-21 09:21:23 +05:30
pukkandan 1619ab3e67
Bugfix for ebe1b4e34f 2023-06-21 09:21:22 +05:30
pukkandan ad54c9130e
[cleanup] Misc
Closes #6288, Closes #7197, Closes #7265, Closes #7353, Closes #5773
Authored by: mikf, freezboltz, pukkandan
2023-06-21 09:21:20 +05:30
pukkandan ebe1b4e34f
[outtmpl] Fix some minor bugs
Closes #7164
2023-06-21 06:10:39 +05:30
pukkandan eedda5252c
[utils] FormatSorter: Improve size and br
Closes #1596

Previously, when some formats have accurate size and some approximate,
the ones with accurate size was always prioritized

For formats with known tbr and unknown vbr/abr, we were setting
(vbr=tbr, abr=0) for sorting to work. This is no longer needed.

Authored by pukkandan, u-spec-png
2023-06-21 06:10:38 +05:30
pukkandan 01aba2519a
[jsinterp] Fix global object extraction
Closes #7327
2023-06-18 04:11:15 +05:30
bashonly b4a252fba8
[jsinterp] Fix division (#7279)
* Fixes nsig decryption for Youtube JS player `8c7583ff`

Authored by: bashonly
2023-06-10 22:49:12 +00:00
Matt Broadway b38d4c941d
[cookies] Update for chromium changes (#6897)
Authored by: mbway
2023-05-29 19:21:35 +05:30
coletdjnz daafbf49b3
[core] Support decoding multiple content encodings (#7142)
Authored by: coletdjnz
2023-05-27 10:40:05 +00:00
coletdjnz b87e01c123
[cookies] Move YoutubeDLCookieJar to cookies module (#7091)
Authored by: coletdjnz
2023-05-27 07:08:19 +00:00
coletdjnz 08916a49c7
[core] Improve HTTP redirect handling (#7094)
Aligns HTTP redirect handling with what browsers commonly do and RFC standards. 

Fixes issues afac4caa7d missed.

Authored by: coletdjnz
2023-05-27 07:06:13 +00:00
pukkandan 7aeda6cc9e
[jsinterp] Do not compile regex 2023-05-24 23:30:45 +05:30
pukkandan 4823ec9f46
Update to ytdl-commit-d1c6c5
[YouTube] [core] Improve platform debug log, based on yt-dlp
d1c6c5c4d6

Except:
    * 6ed34338285f722d0da312ce0af3a15a077a3e2a [jsinterp] Add short-cut evaluation for common expression
        * There was no performance improvement when tested with https://github.com/ytdl-org/youtube-dl/issues/30641
    * e8de54bce50f6f77a4d7e8e80675f7003d5bf630 [core] Handle `/../` sequences in HTTP URLs
        * We plan to implement this differently
2023-05-24 23:30:43 +05:30
pukkandan 6f2287cb18
[cleanup] Misc
Closes #7030, closes #6967
2023-05-20 04:23:41 +05:30
pukkandan 1d7656184c
[jsinterp] Handle NaN in bitwise operators
Closes #6131
2023-05-20 04:07:17 +05:30
Simon Sawicki b079c26f0a
[utils] traverse_obj: More fixes (#6959)
- Fix result when branching with `traverse_string`
- Fix `slice` path on `dict`s
- Fix tests and docstrings from 21b5ec86c2
- Add `is_iterable_like` helper function

Authored by: Grub4K
2023-04-30 19:50:22 +02:00
pukkandan 7cf51f2191
[jsinterp] Handle negative numbers better
Closes #6131
2023-04-27 07:52:09 +05:30
Simon Sawicki 21b5ec86c2
[utils] traverse_obj: Allow iterables in traversal (#6902)
Authored by: Grub4K
2023-04-24 19:56:35 +02:00
pukkandan ec9311c41b
[outtmpl] Support str.format syntax inside replacements
Closes #6843
2023-04-24 18:43:54 +05:30
pukkandan 78fde6e339
[outtmpl] Allow \n in replacements and default.
Fixes: https://github.com/yt-dlp/yt-dlp/issues/6808#issuecomment-1510055357
Fixes: https://github.com/yt-dlp/yt-dlp/issues/6808#issuecomment-1510363645
2023-04-24 18:28:30 +05:30
Simon Sawicki 0898c5c8cc
[utils] js_to_json: Implement template strings (#6623)
Authored by: Grub4K
2023-03-25 19:41:28 +01:00
pukkandan f68434cc74
[extractor] Extract more metadata from ISM
Fixes 81b6102d20 (r105892531)
2023-03-25 13:18:21 +05:30
pukkandan 9acf1ee25f
[jsinterp] Handle Date at epoch 0
Closes #6400
2023-03-03 16:55:06 +05:30
pukkandan 65f6e80780
[dependencies] Simplify Cryptodome
Closes #6292, closes #6272, closes #6338
2023-02-28 23:15:13 +05:30
pukkandan 45b2ee6f4f
Update to ytdl-commit-2dd6c6e
[YouTube] Avoid crash if uploader_id extraction fails
2dd6c6edd8

Except:
    * 295736c9cba714fb5de7d1c3dd31d86e50091cf8 [jsinterp] Improve parsing
    * 384f632e8a9b61e864a26678d85b2b39933b9bae [ITV] Overhaul ITV extractor
    * 33db85c571304bbd6863e3407ad8d08764c9e53b [feat]: Add support to external downloader aria2p
2023-02-17 17:52:23 +05:30
Simon Sawicki 6839ae1f6d
[utils] traverse_obj: Fix more bugs
and cleanup uses of `default=[]`

Continued from b1bde57bef
2023-02-10 19:36:55 +05:30
Simon Sawicki b1bde57bef
[utils] traverse_obj: Fix several behavioral problems
See #6180 for further info

Authored by: Grub4K
2023-02-08 04:11:08 +01:00
pukkandan f6a765ceb5
[dependencies] Standardize Cryptodome imports 2023-02-08 07:28:46 +05:30
Simon Sawicki 776995bc10
[utils] traverse_obj: Various improvements
- Add `set` key for transformations/filters
- Add `re.Match` group names
- Fix behavior for `expected_type` with `dict` key
- Raise for filter function signature mismatch in debug

Authored by: Grub4K
2023-02-02 06:40:19 +01:00
pukkandan 8b008d6254
[jsinterp] Support if statements
Closes #6131
2023-02-01 09:40:16 +05:30
Simon Sawicki 773c272d66
Fix config locations (#5933)
Bug in 8e40b9d1ec
Closes #5953

Authored by: Grub4k, coletdjnz, pukkandan
2023-01-07 00:31:00 +05:30
Matthew 8e40b9d1ec
Improve plugin architecture (#5553)
to make plugins easier to develop and use:
* Plugins are now loaded as namespace packages.
* Plugins can be loaded in any distribution of yt-dlp (binary, pip, source, etc.).
* Plugin packages can be installed and managed via pip, or dropped into any of the documented locations.
* Users do not need to edit any code files to install plugins.
* Backwards-compatible with previous plugin architecture.

As a side-effect, yt-dlp will now search in a few more locations for config files.

Closes https://github.com/yt-dlp/yt-dlp/issues/1389

Authored by: flashdagger, coletdjnz, pukkandan, Grub4K
Co-authored-by: Marcel <flashdagger@googlemail.com>
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
2023-01-01 04:29:22 +00:00
pukkandan fbb7383306
Add weba to known extensions 2022-12-30 15:32:47 +05:30
ChillingPepper d5f043d127
[utils] js_to_json: Fix bug in f55523c (#5771)
Authored by: ChillingPepper, pukkandan
2022-12-30 12:08:38 +05:30
pukkandan 9f14daf22b
[extractor] Deprecate _sort_formats 2022-11-17 11:40:17 +05:30
pukkandan 6368e2e639
[cleanup] Misc
Closes #5541
2022-11-16 06:57:07 +05:30
Robert Geislinger 7aaf4cd2a8
[cleanup] Misc
Closes #5471, Closes #5312

Authored by: pukkandan, Alienmaster
2022-11-11 15:48:29 +05:30
pukkandan 495322b95b
[test] Allow extract_flat in download tests
Authored by: coletdjnz, pukkandan
2022-11-10 07:32:35 +05:30
Matthew 86973308cd
[extractor/youtube:tab] Update tab handling for redesign (#5439)
Closes #5432, #5430, #5419
Authored by: coletdjnz, pukkandan
2022-11-09 14:28:44 +05:30
pukkandan 46d09f8707
[cleanup] Lint and misc cleanup 2022-11-07 02:32:36 +05:30
pukkandan db4678e448
Update to ytdl-commit-de39d128
[extractor/ceskatelevize] Back-port extractor from yt-dlp
de39d1281c

Closes #5361, Closes #4634, Closes #5210
2022-11-07 02:18:30 +05:30
Alex Karabanov c9bd65185c
[extractor/zenyandex] Fix extractors (#3750, #5268)
Closes #3736
Authored by:  lksj, puc9, pukkandan

Co-authored-by: puc9 <51006296+puc9@users.noreply.github.com>
2022-10-25 15:50:48 +05:30
Ajay Ramachandran 63c547d71c
[SponsorBlock] Support chapter category (#5260)
Authored by: ajayyy, pukkandan
2022-10-18 22:21:57 +05:30
Simon Sawicki a71b812f53
[utils] js_to_json: Improve escape handling (#5217)
Authored by: Grub4K
2022-10-13 01:52:17 +05:30
Simon Sawicki 36069409ec
[cookies] Improve LenientSimpleCookie (#5195)
Closes #5186 
Authored by: Grub4K
2022-10-11 09:09:12 +05:30
pukkandan 0468a3b325
[jsinterp] Improve separating regex
Fixes https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1273974909
2022-10-11 08:02:26 +05:30