Commit graph

221 commits

Author SHA1 Message Date
pukkandan 1ad047d0f7
[nebula] Move to nebula.app
Closes #272
Tested by: Lamieur
2021-04-28 19:18:06 +05:30
pukkandan be6202f12b
Subtitle extraction from streaming media manifests #247
Authored by fstirlitz
Modified from: https://github.com/ytdl-org/youtube-dl/pull/6144

Closes: #73
Fixes:
https://github.com/ytdl-org/youtube-dl/issues/6106
https://github.com/ytdl-org/youtube-dl/issues/14977
https://github.com/ytdl-org/youtube-dl/issues/21438
https://github.com/ytdl-org/youtube-dl/issues/23609
https://github.com/ytdl-org/youtube-dl/issues/28132

Might also fix (untested):
https://github.com/ytdl-org/youtube-dl/issues/15424
https://github.com/ytdl-org/youtube-dl/issues/18267
https://github.com/ytdl-org/youtube-dl/issues/23899
https://github.com/ytdl-org/youtube-dl/issues/24375
https://github.com/ytdl-org/youtube-dl/issues/24595
https://github.com/ytdl-org/youtube-dl/issues/27899

Related:
https://github.com/ytdl-org/youtube-dl/issues/22379
https://github.com/ytdl-org/youtube-dl/pull/24517
https://github.com/ytdl-org/youtube-dl/pull/24886
https://github.com/ytdl-org/youtube-dl/pull/27215

Notes:
* The functions `extractor.common._extract_..._formats` are still kept for compatibility
* Only some extractors have currently been moved to using `_extract_..._formats_and_subtitles`
* Direct subtitle manifests (without a master) are not supported and are wrongly identified as containing video formats
* AES support is untested
* The fragmented TTML subtitles extracted from DASH/ISM are valid, but are unsupported by `ffmpeg` and most video players
    * Their XML fragments can be dumped using `ffmpeg -i in.mp4 -f data -map 0 -c copy out.ttml`.
        Once the unnecessary headers are stripped out of this, it becomes a valid self-contained ttml file
    * The ttml subs downloaded from DASH manifests can also be directly opened with <https://github.com/SubtitleEdit>
* Fragmented WebVTT files extracted from DASH/ISM are also unsupported by most tools
    * Unlike the ttml files, the XML fragments of these cannot be dumped using `ffmpeg`
    * The webtt subs extracted from DASH can be parsed by <https://github.com/gpac/gpac>
    * But validity of the those extracted from ISM are untested
2021-04-28 19:02:43 +05:30
Felix S e8f834cd8d [threeqsdn] Extract subtitles from streaming manifests 2021-04-28 17:24:50 +05:30
Felix S e0e624ca7f [canvas] Extract subtitles from streaming manifests 2021-04-28 17:24:19 +05:30
Felix S ec4f374c05 [wat] Extract subtitles from streaming manifests 2021-04-28 17:24:08 +05:30
Felix S c811e8d8bd [atresplayer] Extract subtitles from streaming manifests 2021-04-28 17:23:56 +05:30
Felix S b2cd5da460 [francetv] Extract subtitles from the HLS manifest 2021-04-28 17:23:47 +05:30
Felix S 2de3b21e05 [uplynk] Extract subtitles from HLS manifests 2021-04-28 17:23:37 +05:30
Felix S 4bed436371 [twitter] Extract subtitles from HLS manifests 2021-04-28 17:23:27 +05:30
Felix S efe9dba595 [srgssr] Extract subtitles from HLS manifests 2021-04-28 17:23:16 +05:30
Felix S 47f4203dd3 [nytimes] Extract subtitles from HLS manifests 2021-04-28 17:23:05 +05:30
Felix S 015c10aeec [roosterteeth] Use common code for subtitle extraction 2021-04-28 17:22:56 +05:30
Felix S a00d781b73 [elonet] Use common code for subtitle extraction 2021-04-28 17:22:45 +05:30
Felix S 0c541b563f [tv4] Extract subtitles from streaming manifests 2021-04-28 17:22:36 +05:30
Felix S 64a5cf7929 [byutv] Extract subtitles from streaming manifests 2021-04-28 17:22:27 +05:30
Felix S 7a450a3b1c [generic] Extract subtitles from direct SSTR manifest links 2021-04-28 17:22:18 +05:30
Felix S 7de27caf16 [generic] Extract subtitles from direct DASH manifest links 2021-04-28 17:22:07 +05:30
Felix S c26326c1be [generic] Extract subtitles from direct HLS manifest links 2021-04-28 17:21:55 +05:30
Felix S 66a1b8643a [downloader/ism] Support muxing TTML subtitles 2021-04-28 17:21:45 +05:30
Felix S 15828bcf25 [downloader/hls] Handle MPEG-2 PES timestamp overflow 2021-04-28 17:21:35 +05:30
Felix S 333217f43e [downloader/hls] Remove duplicate cues using a sliding window of candidates 2021-04-28 17:21:26 +05:30
Felix S 4a2f19abbd [downloader/hls] Assemble single-file WebVTT subtitles from HLS segments 2021-04-28 17:21:14 +05:30
Felix S f6a1d69a87 [extractor/common] Extend _extract_akamai_formats to also extract subtitle tracks 2021-04-28 17:20:29 +05:30
Felix S fd76a14259 [extractor/common, downloader/ism] Extract SSTR subtitle tracks
_parse_ism_formats was extended into _parse_ism_formats_and_subtitles;
all direct users were updated, though _extract_ism_formats was left
as a compatibility wrapper.

The SSTR downloader was also modified in order to prepare for muxing
subtitle streams, although no support for any subtitle codecs was
added in this commit.
2021-04-28 17:20:20 +05:30
Felix S 171e59edd4 [extractor/common] Extract DASH subtitle tracks
_extract_mpd_formats and _parse_mpd_formats were extended into
_…_formats_and_subtitles; wrappers with old names are provided
for compatibility.
2021-04-28 17:20:11 +05:30
Felix S a0c3b2d5cf [extractor/common] Extract HLS subtitle tracks
_extract_m3u8_formats is renamed to _extract_m3u8_formats_and_subtitles
and extended to handle subtitle tracks instead of skipping them;
a wrapper with the old name is provided for compatibility.

_parse_m3u8_formats is likewise renamed and extended, but without adding
the compatibility wrapper; the test suite is adjusted to test the enhanced
method instead.
2021-04-28 17:19:57 +05:30
Felix S 19bb39202d [extractor/common] Generalise _merge_subtitles
This allows modifying a subtitles dictionary in-place.
2021-04-28 17:19:46 +05:30
Felix S d4553567d2 [downloader/ism] Prevent writing the header again when resuming an interrupted download 2021-04-28 17:19:37 +05:30
Felix S 4d49884c58 [downloader/fragment] Allow persisting extra state when a download is interrupted 2021-04-28 17:19:31 +05:30
Felix S 5873d4ccdd [utils] Improve bug_report_message
Add an optional argument specifying the text that should go before
the message.
2021-04-28 17:19:23 +05:30
Hadi0609 db9a564b6a
[zee5] Fix extraction for some URLs (#279)
Closes: #278
2021-04-28 14:51:54 +05:30
Felix S c72967d5de
[mediasite] Generalize URL pattern (#275)
Authored by: fstirlitz
2021-04-26 17:23:20 +05:30
pukkandan 598d185db1
Fix case sensitivity of format selector
Bug introduced in f8d4ad9ab0
2021-04-26 10:56:56 +05:30
pukkandan b982cbdd0e
[limelight] Obey allow_unplayable_formats 2021-04-26 10:56:55 +05:30
pukkandan 6a04a74e8b
[FormatSort] Fix for when some formats have quality and others don't 2021-04-26 10:56:54 +05:30
pukkandan 88728713c8
Py2 compatibility for FileNotFoundError 2021-04-26 10:56:53 +05:30
CXwudi 6b1d8c1e30
[niconico] Fix title and thumbnail extraction (#273)
Authored by: CXwudi
2021-04-26 08:23:57 +05:30
Ashish 87c3d06271
[Mxplayer] Add MxplayerShowIE (#270)
Authored by: Ashish0804
2021-04-26 08:12:51 +05:30
pukkandan 915f911e36
[utils] Encode URLs in YoutubeDLCookieProcessor
Closes #263
2021-04-24 19:20:07 +05:30
pukkandan cf9d6cfb0c
[tubi] Raise "no video formats" error when video url is empty
Related: #266
2021-04-24 17:52:33 +05:30
pukkandan bbed5763f1
[francetvinfo] Improve video id extraction
Closes #261
2021-04-23 00:01:09 +05:30
pukkandan ca0b91b39e
[version] update :ci skip all 2021-04-22 17:30:36 +05:30
pukkandan e58c22a0f6
[documentation] Fix typos 2021-04-22 16:54:44 +05:30
pukkandan 1bdae7d312
Update to ytdl-commit-7e8b3f9
[youtube] Remove unused code
7e8b3f9439
2021-04-22 16:54:07 +05:30
Felix S a471f21da6
[mildom] Remove proxy (#260)
Closes #251
Makes 2cff495997, ab406a1c0e, #252 obsolete

Authored by: fstirlitz
2021-04-22 16:52:22 +05:30
pukkandan 6efb071135
[BilibiliChannel] Fix pagination
Closes #222

ccca21d7f5
Coauthored by: nao20010128nao, pukkandan
2021-04-22 04:19:33 +05:30
pukkandan a439a3a45c
Improve output template (see desc)
* Objects can be traversed like `%(field.key1.key2)s`
* A number can be added to the field as `%(field+n)s`
* Deprecates `--autonumber-start`
2021-04-22 04:19:33 +05:30
pukkandan 26e2805c3f
Add option --skip-playlist-after-errors
Allows to skip the rest of a playlist after a given number of errors are encountered
2021-04-22 02:16:31 +05:30
pukkandan 3b4775e021
[go] Fix _VALID_URL
Closes #255
2021-04-21 15:43:53 +05:30
pukkandan ab406a1c0e
[mildom] Warn user of proxy 2021-04-21 15:43:22 +05:30