]> git.gir.st - subscriptionfeed.git/log
subscriptionfeed.git
3 years agoexpose hls/dash streams to template
girst [Sat, 19 Sep 2020 18:12:58 +0000 (20:12 +0200)]
expose hls/dash streams to template

note that this can't be used, since no browsers implement native
hls/dash, and any js-polyfills will trigger cors-errors.

3 years ago(re)add subscribe button to channel page
girst [Sat, 19 Sep 2020 18:11:04 +0000 (20:11 +0200)]
(re)add subscribe button to channel page

3 years agoredirect youtube.com/<username>, fix /c/ trailing slash
girst [Sat, 19 Sep 2020 18:05:22 +0000 (20:05 +0200)]
redirect youtube.com/<username>, fix /c/ trailing slash

3 years agorename 'dangerous' blueprint, as its safe
girst [Sat, 19 Sep 2020 15:06:04 +0000 (17:06 +0200)]
rename 'dangerous' blueprint, as its safe

has been running long enough in production now that I'm relatively
confident that it doesn't get blocked by Big Goog' (*knocks on wood*).

3 years agoclean up dead protobuf structures
girst [Sat, 19 Sep 2020 14:57:06 +0000 (16:57 +0200)]
clean up dead protobuf structures

3 years agofix exception in pagination code on &foo=<emptystring>
girst [Sat, 19 Sep 2020 13:56:52 +0000 (15:56 +0200)]
fix exception in pagination code on &foo=<emptystring>

3 years agofix strange exception on reddit api usage
girst [Sat, 19 Sep 2020 13:53:19 +0000 (15:53 +0200)]
fix strange exception on reddit api usage

data.data might be null (maybe if subreddit list is []?)

3 years agodocument perl installation
girst [Sat, 19 Sep 2020 13:50:13 +0000 (15:50 +0200)]
document perl installation

3 years agoredirect /c/<ucid> to /channel/<ucid>
girst [Sat, 19 Sep 2020 13:49:09 +0000 (15:49 +0200)]
redirect /c/<ucid> to /channel/<ucid>

3 years agomove to v3 channel format
girst [Sat, 19 Sep 2020 13:47:48 +0000 (15:47 +0200)]
move to v3 channel format

modeled after
https://github.com/user234683/youtube-local/commit/1ff97bfde1467f7f18d4d2715a9357c41d9e9b8f

3 years agostore length when pinning videos
girst [Sat, 19 Sep 2020 13:25:55 +0000 (15:25 +0200)]
store length when pinning videos

3 years agoadd link to source code
girst [Thu, 17 Sep 2020 11:03:47 +0000 (13:03 +0200)]
add link to source code

3 years agofix pinning the first (oldest) subscription
girst [Thu, 17 Sep 2020 11:03:26 +0000 (13:03 +0200)]
fix pinning the first (oldest) subscription

3 years agoUse new k-anonymity SponsorBlock API; enable by default
girst [Wed, 16 Sep 2020 14:24:56 +0000 (16:24 +0200)]
Use new k-anonymity SponsorBlock API; enable by default

3 years agoimplement /watch video start offset
girst [Wed, 16 Sep 2020 13:42:46 +0000 (15:42 +0200)]
implement /watch video start offset

3 years agouse correct magic for v2 format
girst [Sat, 5 Sep 2020 16:31:46 +0000 (18:31 +0200)]
use correct magic for v2 format

test channel: UC2C_jShtL725hvbm1arSV9w (sort by popular, page2)

reversing of magic:
https://github.com/user234683/youtube-local/blob/master/youtube/channel.py#L47

3 years agolink to snopyta's invidous instance
girst [Fri, 4 Sep 2020 18:27:51 +0000 (20:27 +0200)]
link to snopyta's invidous instance

3 years agofix error on playlist that doesn't even work on youtube.com
girst [Sat, 29 Aug 2020 17:35:55 +0000 (19:35 +0200)]
fix error on playlist that doesn't even work on youtube.com

3 years agoI AM SO SMART -- S-M-R-T!
girst [Sat, 29 Aug 2020 16:22:23 +0000 (18:22 +0200)]
I AM SO SMART -- S-M-R-T!

3 years agofixup 'blockable'
girst [Sat, 29 Aug 2020 10:03:58 +0000 (12:03 +0200)]
fixup 'blockable'

3 years agofix aspect clamping yet again
girst [Sat, 29 Aug 2020 09:56:16 +0000 (11:56 +0200)]
fix aspect clamping yet again

3 years agouse new card.live property in typed_card
girst [Sat, 29 Aug 2020 09:52:07 +0000 (11:52 +0200)]
use new card.live property in typed_card

3 years agofix unavailable endcard
girst [Sat, 29 Aug 2020 09:43:46 +0000 (11:43 +0200)]
fix unavailable endcard

test id: WYjgHLFZMa0

3 years agocomment UU* (user uploads) playlists
girst [Sat, 29 Aug 2020 09:23:41 +0000 (11:23 +0200)]
comment UU* (user uploads) playlists

3 years agomore lenient channel redirection, catch not existing channels
girst [Sat, 29 Aug 2020 09:18:59 +0000 (11:18 +0200)]
more lenient channel redirection, catch not existing channels

3 years agowe're blockable :(
girst [Sat, 29 Aug 2020 09:16:38 +0000 (11:16 +0200)]
we're blockable :(

3 years agoconvert all search result types to use pipe framework
girst [Sat, 29 Aug 2020 09:14:05 +0000 (11:14 +0200)]
convert all search result types to use pipe framework

3 years agomake |G() take multiple keys
girst [Sat, 29 Aug 2020 08:30:58 +0000 (10:30 +0200)]
make |G() take multiple keys

3 years agoimplement 'pipe' framework and test on VIDEO searchresults
girst [Sat, 29 Aug 2020 08:20:34 +0000 (10:20 +0200)]
implement 'pipe' framework and test on VIDEO searchresults

3 years agonew searchresult types, 'no more results' variant
girst [Sat, 29 Aug 2020 07:10:16 +0000 (09:10 +0200)]
new searchresult types, 'no more results' variant

3 years agoadd /results as alias for /search
girst [Wed, 19 Aug 2020 21:51:42 +0000 (23:51 +0200)]
add /results as alias for /search

youtube-url compat

3 years agoadd final / to /channel/ urls in templates
girst [Wed, 19 Aug 2020 17:22:26 +0000 (19:22 +0200)]
add final / to /channel/ urls in templates

TECHDEBT: this should just use url_for!

3 years agomore innertube renderers
girst [Wed, 19 Aug 2020 17:14:35 +0000 (19:14 +0200)]
more innertube renderers

3 years agouse getpass() instead of input()
girst [Wed, 19 Aug 2020 15:53:23 +0000 (17:53 +0200)]
use getpass() instead of input()

input() writes prompt to stdout, so using it in scripts breaks. (WTF)

3 years agoimplement password change
girst [Wed, 19 Aug 2020 15:32:10 +0000 (17:32 +0200)]
implement password change

3 years agofix learning playlists
girst [Tue, 18 Aug 2020 20:05:43 +0000 (22:05 +0200)]
fix learning playlists

3 years agodon't need to patch youtubedown any more
girst [Mon, 17 Aug 2020 19:24:01 +0000 (21:24 +0200)]
don't need to patch youtubedown any more

as of somewhere between Aug 15 00:52 (last unpatched version downloaded)
and Mon, 17 Aug 2020 19:07:08 GMT (HTTP Last-Modified), it is no longer
necessary to modify youtubedown, as jwz has accepted my patch regarding
caller(). The %ciphers hash has an entry of 13 Aug 2020, so that's
probably the date he applied it locally.

3 years agodon't rely on calc(max()) css to get video to display
girst [Mon, 17 Aug 2020 19:08:39 +0000 (21:08 +0200)]
don't rely on calc(max()) css to get video to display

couldn't watch on my firefox-android 68 (but can now :D)

3 years agoraise exception on webhooks errors
girst [Mon, 17 Aug 2020 16:51:17 +0000 (18:51 +0200)]
raise exception on webhooks errors

logs the traceback.

3 years agoclean up v1/v2 channel protobuf handling
girst [Mon, 17 Aug 2020 16:50:21 +0000 (18:50 +0200)]
clean up v1/v2 channel protobuf handling

3 years agofix channel-playlists
girst [Mon, 17 Aug 2020 16:49:59 +0000 (18:49 +0200)]
fix channel-playlists

has no page number

3 years agofix "learning playlist" and playlists in general, more search result types
girst [Mon, 17 Aug 2020 16:48:15 +0000 (18:48 +0200)]
fix "learning playlist" and playlists in general, more search result types

3 years agohandle absent streamingData.formats/adaptiveFormats
girst [Mon, 17 Aug 2020 15:22:20 +0000 (17:22 +0200)]
handle absent streamingData.formats/adaptiveFormats

e.g. on (announced) livestreams/premiere videos

3 years agopaper over exception on dangerous.search
girst [Mon, 17 Aug 2020 14:52:40 +0000 (16:52 +0200)]
paper over exception on dangerous.search

wasn't able to reproduce this.

[2020-08-17 16:47:53,157] ERROR in app: Exception on /search [GET]
Traceback (most recent call last):
  File "/opt/yt/venv/lib64/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/yt/venv/lib64/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/yt/venv/lib64/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/yt/venv/lib64/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/yt/venv/lib64/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/yt/venv/lib64/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/yt/app/dangerous/__init__.py", line 31, in search
    results, extras = prepare_searchresults(yt_results)
  File "/opt/yt/app/common/innertube.py", line 26, in prepare_searchresults
    return parse_result_items(contents)
  File "/opt/yt/app/common/innertube.py", line 169, in parse_result_items
    r, e = parse_result_items(content['content']['verticalListRenderer']['items'])
  File "/opt/yt/app/common/innertube.py", line 136, in parse_result_items
    listget(content.get('viewCountText',{}).get('runs'),0,{}).get('text')), # "1,234 watching"
  File "/opt/yt/app/common/innertube.py", line 12, in listget
    return next(iter(obj[index:]), fallback)
TypeError: 'NoneType' object is not subscriptable

3 years agoignore more advertisements
girst [Sun, 16 Aug 2020 15:17:42 +0000 (17:17 +0200)]
ignore more advertisements

3 years agohacky way to get v1 and v2 channel responses (needs cleanup!)
girst [Sun, 16 Aug 2020 12:45:14 +0000 (14:45 +0200)]
hacky way to get v1 and v2 channel responses (needs cleanup!)

3 years agodon't crash cipher util when youtubedown can't decode cipher
girst [Sat, 15 Aug 2020 21:27:39 +0000 (23:27 +0200)]
don't crash cipher util when youtubedown can't decode cipher

3 years agofix dummy-card 1px offset
girst [Sat, 15 Aug 2020 20:38:19 +0000 (22:38 +0200)]
fix dummy-card 1px offset

old style removed the border, shifting the last row 1px to the right.

3 years agoREADME: add note regarding new channel protobuf
girst [Sat, 15 Aug 2020 20:14:20 +0000 (22:14 +0200)]
README: add note regarding new channel protobuf

3 years agofallback from dangerous/channel (must fix browse_ajax)
girst [Sat, 15 Aug 2020 19:57:00 +0000 (21:57 +0200)]
fallback from dangerous/channel (must fix browse_ajax)

3 years agomake xmlfeed catch all channel subpages
girst [Sat, 15 Aug 2020 19:54:35 +0000 (21:54 +0200)]
make xmlfeed catch all channel subpages

needed for proper invidious/invidition redirect and fallback_route.

3 years agomake csrf verification use http.parser instead of regexp
girst [Sat, 15 Aug 2020 19:18:13 +0000 (21:18 +0200)]
make csrf verification use http.parser instead of regexp

finally removes the unecessary csrf field from GET forms

3 years agodocument why we aren't deleting videos, clean up logging
girst [Sat, 15 Aug 2020 12:25:43 +0000 (14:25 +0200)]
document why we aren't deleting videos, clean up logging

3 years agosometimes videoDetails.lengthSeconds=="0"
girst [Fri, 14 Aug 2020 16:20:55 +0000 (18:20 +0200)]
sometimes videoDetails.lengthSeconds=="0"

microformat.playerMicroformatRenderer.lengthSeconds might still be good
then. seen on 'premiere' videos.

3 years agomake view count thousand seperator a bit larger
girst [Sun, 9 Aug 2020 10:21:40 +0000 (12:21 +0200)]
make view count thousand seperator a bit larger

3 years agofound new innertube edge case
girst [Sun, 9 Aug 2020 10:21:01 +0000 (12:21 +0200)]
found new innertube edge case

3 years agoRevert "secondary channel format"
girst [Sun, 9 Aug 2020 10:19:15 +0000 (12:19 +0200)]
Revert "secondary channel format"

useless if it doesn't work all the time

This reverts commit 4e3b56b967570b9498651cf758a5ca3c56640689.

3 years agosecondary channel format
girst [Sun, 9 Aug 2020 10:17:38 +0000 (12:17 +0200)]
secondary channel format

reversed by omarroth on #invidious. his message:
The channel you sent me uses a different format. YouTube uses:
```json
{
  "80226972:0:embedded": {
    "2:0:string": "UC6Om9kAkl32dWlDSNlDS9Iw",
    "3:1:base64": {
      "2:0:string": "videos",
      "3:1:varint": 3,
      "4:2:varint": 0,
      "6:3:varint": 1,
      "7:4:varint": 1,
      "61:5:base64": {
        "1:0:embedded": {
          "2:1:embedded": {
            "1:0:base64": {
              "1:0:varint": 120
            }
          }
        }
      }
    }
  }
}
```

where `"1:0:varint": 120` is the offset into the channel.

3 years agoshow LIVE badge on subscription feed
girst [Sat, 8 Aug 2020 21:37:18 +0000 (23:37 +0200)]
show LIVE badge on subscription feed

3 years agoadd livestream detection for subscriptions
girst [Sat, 8 Aug 2020 21:13:35 +0000 (23:13 +0200)]
add livestream detection for subscriptions

note that we usually fetch get_video_info so fast that even for
unannounced live streams, isLive will be false (with isUpcoming true).

3 years agomake version closer to youtube
girst [Sat, 8 Aug 2020 18:54:00 +0000 (20:54 +0200)]
make version closer to youtube

don't think it makes a difference compat-wise (invidious only has
"2.<hardcoded-date>")

3 years agoremove stray end tag
girst [Sat, 8 Aug 2020 16:45:21 +0000 (18:45 +0200)]
remove stray end tag

3 years agomove search css into a stylesheet
girst [Sat, 8 Aug 2020 16:40:02 +0000 (18:40 +0200)]
move search css into a stylesheet

3 years agofix card macro behavriour w.r.t empty channel_id
girst [Sat, 8 Aug 2020 16:17:39 +0000 (18:17 +0200)]
fix card macro behavriour w.r.t empty channel_id

3 years agofix writing youtubedown (unicode chars)
girst [Fri, 7 Aug 2020 19:40:25 +0000 (21:40 +0200)]
fix writing youtubedown (unicode chars)

the cron job runs in LANG=C, so python defaults to ASCII encoding. also
redownload youtubedown if the on-disk size if 0 (e.g. due to this bug)

3 years agocorrect statement about idempotency of cronjobs
girst [Fri, 7 Aug 2020 19:22:20 +0000 (21:22 +0200)]
correct statement about idempotency of cronjobs

3 years agodisplay length on /feed/subscriptions and use new infocard renderer
girst [Fri, 7 Aug 2020 19:21:28 +0000 (21:21 +0200)]
display length on /feed/subscriptions and use new infocard renderer

3 years agoremove broken logging statement
girst [Fri, 7 Aug 2020 19:20:44 +0000 (21:20 +0200)]
remove broken logging statement

3 years agofetch length for new subscription videos
girst [Fri, 7 Aug 2020 19:19:19 +0000 (21:19 +0200)]
fetch length for new subscription videos

in the future, we'll also detect whether it's a livestream

3 years agoimplement video badge (for length)
girst [Fri, 7 Aug 2020 16:23:53 +0000 (18:23 +0200)]
implement video badge (for length)

3 years agomore innertube variants
girst [Fri, 7 Aug 2020 15:59:57 +0000 (17:59 +0200)]
more innertube variants

3 years agosupport playlist end-of-list
girst [Fri, 7 Aug 2020 15:27:25 +0000 (17:27 +0200)]
support playlist end-of-list

3 years agoadd playlist metadata
girst [Fri, 7 Aug 2020 15:11:38 +0000 (17:11 +0200)]
add playlist metadata

3 years agoRevert "useless youtube.com cargoculting"
girst [Fri, 7 Aug 2020 14:54:51 +0000 (16:54 +0200)]
Revert "useless youtube.com cargoculting"

This reverts commit d3b80d96aec748b19f08e8757ae847ce66b4df68.

3 years agouseless youtube.com cargoculting
girst [Fri, 7 Aug 2020 14:54:23 +0000 (16:54 +0200)]
useless youtube.com cargoculting

tried to reduce error ajax responses, but has no effect.

3 years agotone down channel search
girst [Fri, 7 Aug 2020 14:53:32 +0000 (16:53 +0200)]
tone down channel search

3 years agoadd /user and /c redirects for dangerous blueprint
girst [Fri, 7 Aug 2020 14:52:48 +0000 (16:52 +0200)]
add /user and /c redirects for dangerous blueprint

3 years agoadd /c/<username> route (alias for /user/)
girst [Fri, 7 Aug 2020 11:19:35 +0000 (13:19 +0200)]
add /c/<username> route (alias for /user/)

3 years agoimplement channel search
girst [Fri, 7 Aug 2020 11:15:38 +0000 (13:15 +0200)]
implement channel search

3 years agoretry browse_ajax twice on error
girst [Fri, 7 Aug 2020 10:57:04 +0000 (12:57 +0200)]
retry browse_ajax twice on error

3 years agoadd has_more, cleanup, test for error-alerts
girst [Fri, 7 Aug 2020 10:49:29 +0000 (12:49 +0200)]
add has_more, cleanup, test for error-alerts

3 years agodangerous.channel/playlist: parse extra messages
girst [Thu, 6 Aug 2020 21:02:12 +0000 (23:02 +0200)]
dangerous.channel/playlist: parse extra messages

3 years agomove innertube stuff out of dangerous
girst [Thu, 6 Aug 2020 20:59:14 +0000 (22:59 +0200)]
move innertube stuff out of dangerous

3 years agosearch results: fix channel without subscriber count, add spelling info
girst [Thu, 6 Aug 2020 20:38:33 +0000 (22:38 +0200)]
search results: fix channel without subscriber count, add spelling info

3 years agoimplement channel and playlist endpoints using innertube
girst [Thu, 6 Aug 2020 20:01:36 +0000 (22:01 +0200)]
implement channel and playlist endpoints using innertube

ZERO testing regarding whether this can get us banned!

3 years agoimplement search params
girst [Tue, 4 Aug 2020 22:58:59 +0000 (00:58 +0200)]
implement search params

sadly, pure-protobuf is a small 3rd-party dependency :(

3 years agoupdate welcome message
girst [Tue, 4 Aug 2020 19:34:34 +0000 (21:34 +0200)]
update welcome message

3 years agolog info/endcard/searchitems errors to file
girst [Tue, 4 Aug 2020 18:01:40 +0000 (20:01 +0200)]
log info/endcard/searchitems errors to file

3 years agobetter what-to-bold strategy in base.j2
girst [Tue, 4 Aug 2020 17:40:17 +0000 (19:40 +0200)]
better what-to-bold strategy in base.j2

3 years agomove parse_metadata to youtube blueprint
girst [Tue, 4 Aug 2020 17:16:45 +0000 (19:16 +0200)]
move parse_metadata to youtube blueprint

3 years agodon't show 'None' if not video age
girst [Tue, 4 Aug 2020 17:07:40 +0000 (19:07 +0200)]
don't show 'None' if not video age

3 years agoreformat innertube, wrapper for parse_*() -> prepare_*()
girst [Tue, 4 Aug 2020 17:02:16 +0000 (19:02 +0200)]
reformat innertube, wrapper for parse_*() -> prepare_*()

3 years agosplit metadata in essential and extended
girst [Tue, 4 Aug 2020 16:06:27 +0000 (18:06 +0200)]
split metadata in essential and extended

extended is only used by /watch, but essential also for
store_video_metadata(). this allows is to move parse_metadata() to
the youtube blueprint.

3 years ago/watch: use largest thumnail for channel cards
girst [Tue, 4 Aug 2020 11:20:01 +0000 (13:20 +0200)]
/watch: use largest thumnail for channel cards

3 years agofirst draft of non-invidious search (needs cleanup!)
girst [Tue, 4 Aug 2020 11:19:41 +0000 (13:19 +0200)]
first draft of non-invidious search (needs cleanup!)

3 years agomove innertube parsing into its own package, add searchresult parser
girst [Tue, 4 Aug 2020 11:12:02 +0000 (13:12 +0200)]
move innertube parsing into its own package, add searchresult parser

3 years agouse all muxed formats for reliability
girst [Tue, 4 Aug 2020 10:45:46 +0000 (12:45 +0200)]
use all muxed formats for reliability

sometimes, a muxed stream is broken (yet still returns 200OK, so can't
easily detect server side). when the browser tries to load the broken
stream and can't play it, it moves on to the next available one.
was able to reproduce with ytid Xs9rnAH3VAQ

3 years agodisplay name for playlist subscriptions in the manager
girst [Mon, 3 Aug 2020 08:50:46 +0000 (10:50 +0200)]
display name for playlist subscriptions in the manager

3 years agoWIP: subscribing to playlist
girst [Sun, 2 Aug 2020 21:56:19 +0000 (23:56 +0200)]
WIP: subscribing to playlist

this adds the possibility to subscribe to playlists. playlists don't
support websub, so they'll be updated via pull-sub only.

Note that pubsubhubbub.appspot.com pretends you can subscribe to the
'full' feeds, but then never send POSTs on update.

3 years agoreturn channel and playlist ids from parse_xml
girst [Sun, 2 Aug 2020 21:27:35 +0000 (23:27 +0200)]
return channel and playlist ids from parse_xml

needed for /user/<> endpoint's card infobar; later useful when
subscribing to playlists

Imprint / Impressum