girst [Sun, 28 May 2023 15:53:52 +0000 (15:53 +0000)]
use other 'params' value
8AEB is apparently used by shorts, and doesn't return viewCount. this
patch makes viewCount optional, but uses the other cargo-culted value
that does return viewCount (for now, at least).
girst [Sun, 28 May 2023 15:31:45 +0000 (15:31 +0000)]
*real* fix for 403 problem
apparently, youtube shorts adds '8AEB', another working parameter might
be 'CgIQBg%3D%3D'. all this is untested and we just cargo-culted 8AEB.
it turns out, this
1) isn't a tcp problem, but just googlevideo blacklisting generated urls
on second/third(ish) reload
2) was known by newpipe and invidious for half a year.
avoiding youtubei.googleapis.com was a red herring, too.
more unsorted information:
https://github.com/TeamNewPipe/NewPipe/issues/9038#issuecomment-1289756816
https://github.com/TeamNewPipe/NewPipeExtractor/pull/960
https://github.com/ytdl-org/youtube-dl/issues/30725
https://github.com/iv-org/invidious/commit/09b9b758dea2e9dc502ba497a34ae02cbb37ae46
NewPipeExtractor writes in YoutubeStreamExtractor.java @a4db106:
// Workaround getting streaming URLs which can return 403 HTTP response
// codes by using stories parameter for Android client requests
// This behavior only happen in certain countries such as UK as of
// 10.29.2022
girst [Wed, 24 May 2023 19:19:25 +0000 (19:19 +0000)]
fix tcp streams aborting
since updating the android parameters, downloading of streams would sometimes
stop. it seems to be correlated to pausing the video, probably the browser
stops further buffering and resuming the download then fails.
these parameters are taken from newpipe (sdk version, user agent, app version)
not sure if using the correct api key helps, but doing this now anyways.
interestingly, switching the api host name to youtubei.googleapis.com (which
is what supposedly the android app uses) reintroduces the stream bug.
girst [Wed, 3 May 2023 19:30:28 +0000 (19:30 +0000)]
fix bug in shorts detection logic
a (potential) short was marked as False instead of None either of these situations:
- length < 60 and aspect_ratio is None
- length is None and aspect_ratio < 1
replaces VERY HEAVY SALTIRE from the SMP with MULTIPLICATION X from the
BMP. the closer equivalent would have been HEAVY MULTIPLICATION X, but I
like the sleeker look. this was done for client compatibility;
especially mobile browsers lack the former glyph.
livestreams currently live and finished livestreams will show 'LIVE'.
the latter is a limitation of our collected data (at the time of
crawling, livestreams have no duration which we could show).
use redirect.invidious.io instead of hard-coding a fallback-instance.
note that some instances provide a redirect feature; this would be cool
to have on redirect. or api.invidious.io, but it isn't.
https://invidious.snopyta.org/redirect?referer=%2Fwatch%3Fv%3D{video_id}
removed from reddit completely (who even used that?).
subscription feed: filter shorts if the user enabled the 'noshorts' setting
videos are displayed iff either noshorts config is false or not a shorts
video, but pinning overrides hiding. if shorts are shown, they are
marked 'shorts' instead of the length.
[DATABASE CHANGE: Migration below] allow setting user settings from profile page
CREATE TABLE IF NOT EXISTS user_settings(
user_id INTEGER,
setting TEXT NOT NULL,
value TEXT NOT NULL,
PRIMARY KEY(user_id, setting),
FOREIGN KEY(user_id) REFERENCES users(id));
cargo-cults some additional parameters. the following ones are sometimes
included, but don't seem to have an effect:
"user": {
"lockedSafetyMode": False,
},
"playbackContext": {
"contentPlaybackContext": {
"html5Preference": "HTML5_PREF_WANTS",
},
},
previously, we used a mix of invidious, newpipe, ytdlp,
zerodytrash/YouTube-Internal-Clients values for no particular reason.
this should get us less distinct (we still are using the wrong api key).
channels with a very small amount of videos return a onResponseReceivedActions
array with just 1 element instead of two. normally, [1] has the videos, and [0]
has a feedFilterChipBarRenderer object, containing continuation tokens for
"Latest" and "Popular".
examples: UCx6cailiCkg_mlMM7JX5yfA and UCqqnIXYyT9t3as2ZBisN33g
listing all urls in the <video> element doesn't work (in firefox at
least), so only load highest resolution one by default. we can then just
override that choice when a &itag=XX is present.
girst [Sat, 8 Jan 2022 23:11:44 +0000 (00:11 +0100)]
remove 2nd no-url bailout again
this replaces 18bf5ede; the problem was that without url, the call to
urlparse() would fail. bailing out at this point however breaks
livestreams (and premieres). the real fix is of course to not check for
geolocking if there's no url.
girst [Sat, 8 Jan 2022 00:06:36 +0000 (01:06 +0100)]
finally good /c/ and /user/ matching
don't look up both types of channels and return just one, but do the
correct lookup in the first place. no point in doing the fallback lookup
now that i'm fairly certain the lookup does the right thing.
note: using @app.route('/<any(user, c):typ>/<user>/') doesn't work,
probably because it then doesn't override the hardcoded rule in
youtube/.
girst [Wed, 5 Jan 2022 21:42:10 +0000 (22:42 +0100)]
ignore hashtag related search results
includingResultsForRenderer is shown when searching e.g. for '#foo' to
also show results for 'foo'. hashtagTileRenderer is a link to
'/hashtag/<search_query>' telling you that there are X videos from X
channels about that topic.
girst [Wed, 5 Jan 2022 21:09:24 +0000 (22:09 +0100)]
fix COURSE/"learning playlist" special-casing
this appears to no longer be needed. new test case (since the tomscott
playlist can't be searched for any more):
- PLRqwX-V7Uu6Zy51Q-x9tMWIv9cueOFTFA
"Code! Programming with p5.js" by "The Coding Train"
- /search?q=The+Coding+Train&type=playlist
- /channel/UCvjgXvBlbQiydffZU7m1_aw/playlists
girst [Sat, 20 Nov 2021 13:33:27 +0000 (14:33 +0100)]
try /c/ before /user/ when canonicalizing user names
especially for <instance_host>/<something>; this returned the wrong
channel, if there is both a /c/ and a /user/ with the same id.
example:
c/BoyBoyProductions => UC_S45UpAYVuc0fYEcHN9BVQ (would be correct)
user/BoyBoyProductions => UC2ct7L3mSnprF5o0GAWLzrg (this was selected)
girst [Sat, 20 Nov 2021 13:14:59 +0000 (14:14 +0100)]
allow limiting proxy to authenticated users
g.proxy_on should be accessed through getattr(g, 'proxy_on', None).
This allows for the following states:
True: proxy can be used
False: proxy use is denied; try logging in
None: proxy is disabled globally