mirrors/yt-dlp
1
0
Fork 0
mirror of https://github.com/yt-dlp/yt-dlp.git synced 2024-11-16 05:13:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
20433 commits 3 branches 99 tags 136 MiB
Commit graph

262 commits

Author SHA1 Message Date
Yen Chi Hsuan 7dc2a74e0a
[utils] Fix unified_timestamp for formats parsed by parsedate_tz() 2016-08-05 11:41:55 +08:00
Yen Chi Hsuan 0b68de3cc1 Merge pull request #8876 from remitamine/html5_media 
[extractor/common] add helper method to extract html5 media entries
 2016-07-10 23:40:45 +08:00
Yen Chi Hsuan 84c237fb8a
[utils] Add get_element_by_class 
For #9950
 2016-07-06 20:02:52 +08:00
Remita Amine dfaa86b75e [test_utils] add test for smuggling a smuggled url 2016-07-04 21:36:32 +01:00
remitamine 4f3c5e0627 [utils] add helper function for parsing codecs 2016-06-26 14:03:58 +01:00
Yen Chi Hsuan 1143535d76
[utils] Add urshift() 
Used in IqiyiIE and LeIE
 2016-06-26 15:16:49 +08:00
Sergey M․ 46f59e89ea
[utils] Add unified_timestamp 2016-06-25 23:19:18 +07:00
Yen Chi Hsuan 47212f7bcb
[utils] Don't transform numbers not starting with a zero 
Fix test_Viidea and maybe others
 2016-06-16 11:00:54 +08:00
Yen Chi Hsuan 55b2f099c0
[utils] Decode HTML5 entities 
Used in test_Vporn_1. Also related to #9270
 2016-06-10 15:11:55 +08:00
bzc6p b96f007eeb Added sanitization support for Hungarian letters Ő and Ű 2016-06-02 11:39:32 +02:00
Sergey M․ 46bc9b7d7c
[utils] Allow None in remove_{start,end} 2016-05-19 04:31:30 +06:00
Sergey M․ 364cf465dd
[test_utils] PEP 8 2016-05-14 20:46:33 +06:00
Sergey M․ 89ac4a19e6
[utils] Process non-base 10 integers in js_to_json 2016-05-14 20:39:58 +06:00
felix bd1e484448
[utils] js_to_json: various improvements 
now JS object literals like { /* " */ 0: ",]\xaa<\/p>", } will be correctly converted to JSON.
 2016-05-14 20:12:39 +06:00
Yen Chi Hsuan 778a1ccca7
[utils] Add Œ and œ found in French to ACCENT_CHARS 
Fixes #9463
 2016-05-12 19:48:48 +08:00
Yen Chi Hsuan dab0daeeb0
[utils,compat] Move struct_pack and struct_unpack to compat.py 2016-05-10 14:51:38 +08:00
Adam Thalhammer 31c4448f6e Instead of replacing accented characters with an underscore when sanitizing file names in restricted mode, replace them with their non-accented equivalents fixes #9347 2016-05-02 13:25:12 +10:00
Adam Thalhammer 79a2e94e79 Instead of replacing accented characters with an underscore when sanitizing file names in restricted mode, replace them with their non-accented equivalents fixes #9347 2016-05-02 13:21:39 +10:00
Sergey M b6c0d4f431 Merge pull request #9110 from remitamine/parse_duration 
[utils] imporove parse_duration to handle more formats
 2016-04-21 22:53:16 +07:00
remitamine acaff49575 [utils] imporove parse_duration to handle more formats 2016-04-21 16:34:54 +01:00
Jaime Marquínez Ferrándiz eb9c3edd5e [test/utils] Add test for date_from_str 2016-04-09 22:40:05 +02:00
Yen Chi Hsuan 81f36eba88 [test/test_utils] Update for escape_url change (again) 2016-03-23 23:23:26 +08:00
Yen Chi Hsuan 2d60465e44 [test/test_utils] Update for escape_url change 2016-03-23 23:20:28 +08:00
Jaime Marquínez Ferrándiz 782b1b5bd1 [utils] lookup_unit_table: Match word boundary instead of end of string 2016-03-19 11:44:49 +01:00
Sergey M․ c5229f3926 [utils] PEP 8 2016-03-16 21:50:04 +06:00
remitamine 83548824c2 Merge pull request #8092 from bpfoley/twitter-thumbnail 
[utils] Add extract_attributes for extracting html tag attributes
 2016-03-16 13:16:27 +01:00
Sergey M․ fb47597b09 [bbc] Generalize unit table lookup and add parse_count 2016-03-13 16:27:20 +06:00
remitamine 3201a67f61 [test/test_utils] add more tests for update_url_query 2016-03-03 19:18:57 +01:00
remitamine fb640d0a3d [test/test_utils] add tests for update_url_query 2016-03-03 18:40:05 +01:00
Brian Foley 8bb56eeeea [utils] Add extract_attributes for extracting html tag attributes 
This is much more robust than just using regexps, and handles all
the common scenarios, such as empty/no values, repeated attributes,
entity decoding, mixed case names, and the different possible value
quoting schemes.
 2016-03-03 10:11:37 +00:00
Yen Chi Hsuan 5eb6bdced4 [utils] Multiple changes to base_n() 
1. Renamed to encode_base_n()
2. Allow tables longer than 62 characters
3. Raise ValueError instead of AssertionError for invalid input data
4. Return the first character in the table instead of '0' for number 0
5. Add tests
 2016-02-27 03:22:52 +08:00
Sergey M․ f160785c5c [utils] Remove AM/PM from unified_strdate patterns 2016-02-25 00:52:49 +06:00
Yen Chi Hsuan 5bc880b988 [utils] Add OHDave's RSA encryption function 2016-02-20 19:54:58 +08:00
Sergey M․ 8411229bd5 [utils] Allow dot in strip_jsonp 2016-02-07 19:47:09 +06:00
Sergey M․ 86296ad2cd [utils] Add ability to control skipping false values in dict_get 2016-02-07 08:13:04 +06:00
Sergey M․ cbecc9b903 [utils] Add dict_get convenience method 2016-02-07 06:12:53 +06:00
Sergey M․ 6b77d52b1f [test_utils] Add tests for encode_compat_str 2015-12-20 07:07:14 +06:00
Yen Chi Hsuan db2fe38b55 [utils] Support alternative timestamp format in TTML 
Fixes #7608
 2015-12-19 19:29:51 +08:00
Yen Chi Hsuan d631d5f9f2 [utils] Fix TTML conversion 
Tolerate invalid timestamps (closes #7909)
 2015-12-19 18:21:42 +08:00
Sergey M․ 31b2051e21 [utils] Add remove_quotes 2015-12-14 21:30:58 +06:00
Sergey M․ 9cb9a5df77 [utils] Check ext with trailing slash against the list of known extensions 2015-11-22 17:27:13 +06:00
Sergey M․ 5035536e3f [test_utils] Add tests for determine_ext 2015-11-22 06:33:52 +06:00
Sergey M․ 7aefc49c40 [utils] Skip invalid/non HTML entities (Closes #7518) 2015-11-16 20:20:16 +06:00
Jaime Marquínez Ferrándiz 6a75040278 [utils] unified_strdate: Return None if the date format can't be recognized (fixes #7340) 
This issue was introduced with ae12bc3ebb, it returned 'None'.
 2015-11-02 14:08:38 +01:00
Sergey M 30eecc6a04 Merge pull request #7296 from jaimeMF/xml_attrib_unicode 
Use a wrapper around xml.etree.ElementTree.fromstring in python 2.x (…
 2015-10-31 18:15:21 +00:00
Sergey M․ 578c074575 [utils] Support list of xpath in xpath_element 2015-10-31 22:39:44 +06:00
Sergey M․ 52c3a6e49d [utils] Improve parse_iso8601 2015-10-28 21:40:22 +06:00
Jaime Marquínez Ferrándiz 36e6f62cd0 Use a wrapper around xml.etree.ElementTree.fromstring in python 2.x (#7178) 
Attributes aren't unicode objects, so they couldn't be directly used in info_dict fields (for example '--write-description' doesn't work with bytes).
 2015-10-25 20:13:16 +01:00
Sergey M․ d01949dc89 [utils:js_to_json] Fix bad escape in double quoted strings 2015-10-20 23:09:51 +06:00
Sergey M․ f71264490c [test_utils] Add tests for cli option converters 2015-09-05 03:07:19 +06:00
Sergey M․ 87f70ab39d [test_utils] Add more tests for xpath 2015-09-05 00:36:16 +06:00
Sergey M․ ee114368ad [utils] Make value optional for find_xpath_attr 
This allows selecting particular attributes by name but without specifying the value and similar to xpath syntax `[@attrib]`
 2015-08-01 20:22:13 +06:00
Yen Chi Hsuan 9c29bc69f7 [utils] Improve parse_duration 
Now dots are parsed. For example '87 Min.'
 2015-07-22 23:15:22 +08:00
Yen Chi Hsuan 1b0427e6c4 [utils] Support TTML without default namespace 
In a strict sense such TTML is invalid, but Yahoo uses it.
 2015-05-19 00:45:01 +08:00
Yen Chi Hsuan 7dff03636a [utils] Support 'dur' field in TTML 2015-05-12 12:47:37 +08:00
Yen Chi Hsuan d39e0f05db [utils] Remove sanitize_url_path_consecutive_slashes() 
This function is used only in SohuIE, which is updated to use a new
extraction logic.
 2015-05-09 17:37:39 +08:00
Yen Chi Hsuan 0fe2ff78e6 [NBC] Enhance embedURL extraction (closes #2549) 2015-05-04 21:55:04 +08:00
Sergey M․ b3ed15b760 [utils] Add replace_extension 2015-05-02 23:23:06 +06:00
Sergey M․ a4bcaad773 [test_utils] Add tests for prepend_extension 2015-05-02 23:10:48 +06:00
Yen Chi Hsuan bf6427d2fb [ffmpeg] Add dfxp (TTML) subtitles support (#3432, #5146) 2015-04-25 23:18:27 +08:00
Yen Chi Hsuan 0a1603634b [utils] Remove url_infer_protocol 2015-04-08 21:39:34 +08:00
Yen Chi Hsuan 418c5cc3fc [udn] Add new extractor 2015-04-08 17:26:51 +08:00
Sergey M․ 8cf70de428 [test_utils] Add test for unified_strdate 2015-04-04 19:11:01 +06:00
Sergey M․ ba9e68f402 [utils] Drop trailing comma before closing brace 2015-04-04 17:48:55 +06:00
Naglis Jonaitis 91757b0f37 [utils] Escape all HTML entities written in hexadecimal form 2015-03-26 17:15:27 +02:00
Jaime Marquínez Ferrándiz 5379a2d40d [test/utils] Test xpath_text 2015-03-21 14:12:43 +01:00
Sergey M․ 92a4793b3c [utils] Place sanitize url function near other sanitizing functions 2015-03-17 21:34:22 +06:00
Sergey M․ dc03a42537 Merge branch 'sohu_fix' of https://github.com/yan12125/youtube-dl into yan12125-sohu_fix 2015-03-17 21:18:36 +06:00
Sergey M․ 2ebfeacabc [utils] Keep dot and dotdot unmodified (Closes #5171) 2015-03-10 00:50:11 +06:00
Sergey M․ f18ef2d144 [utils] Disallow trailing dot in sanitize_path for a path part 2015-03-08 22:08:48 +06:00
Sergey M․ a2aaf4dbc6 [utils] Add sanitize_path 2015-03-08 20:55:22 +06:00
Yen Chi Hsuan 55969016e9 [utils] Add a function to sanitize consecutive slashes in URLs 2015-03-06 12:43:49 +08:00
Philipp Hagemeister a7440261c5 [utils] Streap leading dots 
Fixes #2865, closes #5087
 2015-03-02 19:07:19 +01:00
Philipp Hagemeister 3e675fabe0 [airmozilla] Be more tolerant when nonessential items are missing (#5030) 2015-02-26 01:25:00 +01:00
Philipp Hagemeister 5a42414b9c [utils] Prevent hyphen at beginning of filename (Fixes #5035) 2015-02-24 11:38:01 +01:00
Philipp Hagemeister d305dd73a3 [utils] Fix js_to_json 
Previously, the runtime could be atrocious for longer inputs.
 2015-02-18 23:59:51 +01:00
Philipp Hagemeister 347de4931c [YoutubeDL] Add generic video filtering (Fixes #4916) 
This functionality is intended to eventually encompass the current format filtering.
 2015-02-10 03:32:24 +01:00
Philipp Hagemeister 9bb8e0a3f9 [wsj] Add new extractor (Fixes #4854) 2015-02-03 10:58:28 +01:00
Philipp Hagemeister 8f4b58d70e [ntvde] Add new extractor (Fixes #4850) 2015-02-02 21:48:54 +01:00
Philipp Hagemeister cfb56d1af3 Add --list-thumbnails 2015-01-25 02:43:19 +01:00
Philipp Hagemeister 61ca9a80b3 [generic] Add support for BOMs (Fixes #4753) 2015-01-23 01:21:30 +01:00
Naglis Jonaitis a69801e2c6 [utils] Add additional format to unified_strdate 2015-01-14 00:16:34 +02:00
Sergey M․ a5fb718c50 [test_utils] Add more tests for parse_duration 2015-01-12 21:39:58 +06:00
Philipp Hagemeister 2aeb06d6dc [utils] Improve colon handling (Fixes #4683) 2015-01-11 17:40:45 +01:00
Philipp Hagemeister 0590062925 Respect age_limit when listing extractors (Fixes #4653) 2015-01-07 07:20:20 +01:00
Philipp Hagemeister cae97f6521 Improve and test ffmpeg version detection 2014-12-14 21:59:59 +01:00
Philipp Hagemeister 42bdd9d051 [cinchcast] Add new extractor (Fixes #4428) 2014-12-12 02:57:36 +01:00
Philipp Hagemeister 47d7c64274 [test_utils] Make test more realistically (#4377) 2014-12-06 12:36:23 +01:00
Philipp Hagemeister 5f9b83944d [ffmpeg] Improve version check and call it from hls (Fixes #4377) 2014-12-06 12:14:26 +01:00
Philipp Hagemeister e8df5cee12 [minhateca] Fix duration parsing 2014-12-04 17:35:40 +01:00
Philipp Hagemeister 4349c07dd7 [minhateca] Add extractor (Fixes #4094) 2014-12-04 17:02:05 +01:00
Philipp Hagemeister e075a44afb [tests] Remove useless u prefixes 2014-11-26 13:07:32 +01:00
Philipp Hagemeister be64b5b098 [xminus] Simplify and extend (#4302) 2014-11-25 09:54:54 +01:00
Jouke Waleson 8bcc875676 PEP8: more applied 2014-11-23 21:20:46 +01:00
Jouke Waleson 5f6a1245ff PEP8 applied 2014-11-23 20:41:03 +01:00
Philipp Hagemeister 7d4111ed14 Provide guidance when called with a YouTube ID starting with a dash. 
Reported at https://news.ycombinator.com/item?id=8648121
 2014-11-23 10:51:09 +01:00
Philipp Hagemeister 6a68bb574a [eporner] Fix duration (Closes #4188) 2014-11-16 14:55:22 +01:00
Philipp Hagemeister 609a61e3e6 [npo] Improve npo.nl (Fixes #4173) 2014-11-13 16:28:05 +01:00
Philipp Hagemeister 4c0924bb24 [utils] Fix intlist_to_bytes in Python 2 (#4181) 2014-11-13 15:28:42 +01:00
Philipp Hagemeister e4bdb37ec6 [spiegel] Add support for embeds 2014-11-13 15:02:31 +01:00
Philipp Hagemeister 12ea2f30cf [utils] Remove unused get_meta_content function 2014-11-04 23:20:39 +01:00
Philipp Hagemeister 8c25f81bee [util] Move compatibility functions out of util 
utils is large enough without these compatibility functions.

Everything that is present in newer versions of Python (i.e. with dev Python it's just an import) goes into compat.py .
Everything else (i.e. youtube-dl-specific helpers) goes into utils.py .
 2014-11-02 11:23:42 +01:00
Philipp Hagemeister 7d11297f3f Merge branch 'master' of github.com:rg3/youtube-dl 2014-10-29 20:10:07 +01:00
Philipp Hagemeister 6ad4013d40 [drtv] Allow fractional timestamps (Fixes #4059) 2014-10-29 20:10:00 +01:00
Jaime Marquínez Ferrándiz 8abec2c8bb [test_utils] Fix compat_getenv and compat_expanduser tests on python 3.x 2014-10-29 11:13:34 +01:00
Sergey M․ f889cea109 Merge branch 'compat-getenv-and-expanduser' of https://github.com/dstftw/youtube-dl into dstftw-compat-getenv-and-expanduser 
Conflicts:
	test/test_utils.py
	youtube_dl/__init__.py
 2014-10-26 19:56:52 +07:00
Philipp Hagemeister e7b6d12254 [utils] Improve and test js_to_json 2014-10-01 00:08:34 +02:00
Philipp Hagemeister 410f3e73ab [utils] Fix js_to_json 2014-10-01 00:08:28 +02:00
Sergey M․ 4644ac5527 [core] Decode environment variables with filesystem encoding (Fixes #3854, Fixes #3217, Fixes #2918) 
Introduces compat versions of os.getenv and os.path.expanduser
 2014-09-30 22:27:53 +07:00
Philipp Hagemeister 99b67fecc5 [arte] Fix upload date extraction 2014-09-29 12:45:18 +02:00
Philipp Hagemeister 9c44d2429b [vimeo:likes] Support large like lists (Fixes #3847) 2014-09-29 00:36:06 +02:00
Philipp Hagemeister 0003a5c416 Merge remote-tracking branch 'dstftw/escape-non-ascii-in-urls' 
Conflicts:
	test/test_utils.py
 2014-09-15 15:40:10 +02:00
Philipp Hagemeister a020a0dc20 [facebook] Fix support for untitled videos (Fixes #3757) 2014-09-15 15:10:24 +02:00
Sergey M․ d05cfe0600 [YoutubeDL/utils] Clarify rationale for URL escaping in comment, move escape routines to utils and add some tests 2014-09-13 20:59:16 +07:00
Sergey M․ ca7b3246b6 [utils] Improve parse_duration 2014-08-31 06:41:30 +07:00
Philipp Hagemeister 4e408e479b [utils] Modernize tests 2014-08-27 19:11:45 +02:00
Philipp Hagemeister 7adcbe7594 [rtlnl] Extract duration 2014-08-25 12:59:53 +02:00
Jaime Marquínez Ferrándiz 173a7026d5 [test/test_utils] Fix typo in method name 2014-08-10 11:08:56 +02:00
Philipp Hagemeister 676eb3f2dd Fix unicode_escape (Fixes #2695) 2014-04-04 23:00:51 +02:00
Philipp Hagemeister fac55558ad [washingtonpost] Add extractor (Fixes #2622) 2014-03-24 23:21:20 +01:00
Philipp Hagemeister 912b38b428 [instagram] Fix info_dict key name 2014-03-24 01:40:09 +01:00
Jaime Marquínez Ferrándiz b74fa8cd2c [facebook] Fix login process 
It was broken and didn't work in python 3.
And use `_download_webpage` instead of `compat_urllib_request.urlopen`.
 2014-03-07 15:25:33 +01:00
Philipp Hagemeister 62e609ab77 Ignore BOM in batch files (Fixes #2450) 2014-02-25 01:43:17 +01:00
Sergey M. 2db806b4aa Improve parse_duration 2014-02-17 03:46:26 +07:00
Philipp Hagemeister b53466e168 Fix f4m downloading on Python 2.6 2014-02-15 16:24:43 +01:00
Philipp Hagemeister 026fcc0495 Fix #2355 (date parsing with dashes) 2014-02-09 18:09:57 +01:00
Philipp Hagemeister 65697b3bf3 Merge branch 'paged-lists' 
Conflicts:
	test/test_utils.py
	youtube_dl/extractor/youtube.py
 2014-01-22 20:00:16 +01:00
Philipp Hagemeister 5aafe895fc Correct XML ampersand fixup 2014-01-20 22:11:34 +01:00
Philipp Hagemeister b7ab059084 Add infrastructure for paged lists 
This commit allows to download pages in playlists as needed instead of all at once.
Before this commit,
    youtube-dl http://www.youtube.com/user/ANNnewsCH/videos --playlist-end 2 --skip-download
took quite some time - now it's almost instantaneous.
As an example, the youtube:user extractor has been converted.
Fixes #2175
 2014-01-20 11:36:47 +01:00
Philipp Hagemeister 608d11f515 [cnn] Add multiple formats, duration, and upload_date 2013-12-26 13:49:44 +01:00
Philipp Hagemeister d6c7a367e8 [utils] Fix url_basename 2013-12-17 12:32:58 +01:00
Philipp Hagemeister 29eb517403 Add webpage_url_basename info_dict field (Fixes #1938) 2013-12-17 04:13:36 +01:00
Jaime Marquínez Ferrándiz f53c966a73 [dailymotion] Extract view count (#1895) 2013-12-06 13:36:36 +01:00
Jaime Marquínez Ferrándiz a6a173c2fd utils.shell_quote: Convert the args to unicode strings 
The youtube test video failed with `UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 34: ordinal not in range(128)`, the problem was with the filenames being encoded.
 2013-11-21 14:09:28 +01:00
Philipp Hagemeister 9d4660cab1 [generic] Support embedded vimeo videos (#1602) 2013-10-15 12:05:13 +02:00
Philipp Hagemeister 44a5f1718a Simplify tests 
* Make them directly executable again
* Move common stuff (md5, parameters) to helper
* Never import *
* General clean up
 2013-10-15 02:00:55 +02:00
Jaime Marquínez Ferrándiz d7e66d39a0 Add an extractor for internetvideoarchive.com videos 
It's used by videodetective.com
 2013-10-12 21:34:04 +02:00
Jaime Marquínez Ferrándiz a921f40799 [ustream] Simplify channel extraction 
the ChannelParser has been moved to a new function in utils get_meta_content
Instead of the SocialStreamParser now it uses a regex
 2013-09-13 22:05:29 +02:00
Philipp Hagemeister 59ae56fad5 Add helper function find_path_attr 2013-07-11 16:12:08 +02:00
Jaime Marquínez Ferrándiz 37254abc36 Allow to use relative dates in the format (now|today)[+-][0-9](day|week|month|year)(s)? (Closes #137) 
Also fix DateRange not accepting ranges of one day.
 2013-04-28 11:39:37 +02:00
Jaime Marquínez Ferrándiz bf50b0383e Fix some IEs that didn't return the uploade_date in the YYYYMMDD format 
Create a function unified_strdate in utils.py to fix these problems
 2013-04-27 15:14:20 +02:00
Jaime Marquínez Ferrándiz bd55852517 Allow to select videos to download by their upload dates (related #137) 
Only absolute dates.
 2013-04-27 14:01:55 +02:00
Philipp Hagemeister 314d506b96 Do not use deprecated method 2012-12-20 13:26:37 +01:00
Philipp Hagemeister e387eb5aba Let youtube IE handle IDs starting with PL (Closes #572) 2012-12-04 10:59:38 +01:00
Philipp Hagemeister 796173d08b Keep video IDs verbatim if possible (Closes #571) 2012-12-03 15:36:41 +01:00
Philipp Hagemeister 627dcfff39 Restrict more characters (Closes #566) 2012-11-28 12:59:27 +01:00
Philipp Hagemeister 59ae15a507 Convert all tabs to 4 spaces (PEP8) 2012-11-28 02:04:46 +01:00
Philipp Hagemeister 8cd10ac4ef Fix printing title etc. 2012-11-28 00:46:21 +01:00
Philipp Hagemeister dae7c920f6 Make test_utils.py run on Python 3 2012-11-27 23:20:29 +01:00
Philipp Hagemeister dffe658bac Remove exclamation mark in --restrict-filenames mode 2012-11-27 17:15:33 +01:00