README.md

   1 # Unnamed Youtube Frontend
   2
   3 [rough functional overview diagram](https://viewer.diagrams.net/?title=youtube-frontend-2020jul19#R7Vxbk5s4Fv41rtl9sIuLuT32Je6d2kylK121k%2BxLSgYZNMHIC6Lbzq9fiZtBkt10m4vpmjx00EEI%2FJ2rjo400%2B%2B2%2B4cY7II%2FsAfDmaZ4%2B5l%2BP9M0R1XoX0Y45ARbWeYEP0ZeTlKPhCf0CxbE4jk%2FRR5MGh0JxiFBuybRxVEEXdKggTjGL81uGxw237oDPhQITy4IReqfyCNB8Ss060j%2FF0R%2BUL5ZNZ38zhaUnYtfkgTAwy81kv5ppt%2FFGJP8aru%2FgyHDrsQlf2514m71YTGMSJsHQuPpYYOtH9%2FX353%2F%2Frj740n1N3PLzId5BmFa%2FOLia8mhhIB%2B%2BI5d0lcRBMKvFGYQ%2Bezu7UuACHzaAZfdf6ECQGkB2Ya0pdLLNSYEb4tGXKDErkO4KS8Jpg%2FdK%2FRqg8LwDoc4pu0IR2z8GKcERf4d%2FYkw%2Fkbpc2VhULoHkgB65cCYAIJwxG47bCARmQKsZxgTuK%2BRCqQeIN5CEh9ol%2BKubeRPFFKrqUX75SgDxrJgbFDjv6YXRFDInV8NfWQNvSi4I%2BfU%2Fb9%2FqU%2F48ctPLfmy0a3%2FfJrHaK7qr3OKwhV5GS7K68yRwN2A9QRkEmBPolgqcYGiaSoCikuzLxSl8m60kPceULxIFnkURRArvOogVvLZOYjlwDXMoEetZtHEMQmwjyMQfjpSb5uoHvt8xpkBYFj9BQk5FC4ApAQ3kU4ItT83zKQfoc5oK8Q%2BPxsWRl7Zww1BkiA3JxZd2DBwj8i3sju9%2Fs6uM6PCWvf72q37Q9E4ycQEp7ELz0BV2g36nT4k5zrm%2FRiOZ2UihiE1ds9N7yTjcPYoBQMcah12GEUkqY38yAg1UbObsmYUfu4oLfmIR9mpPu394lQqxFElQ0wdMAsaAAFrkMCTLol1SAiOW%2FiiLhTRaKKj26IqysyZ2Zci6s4YikgBjA%2FfcrUxynauRk7VPipS1jrUW48wRhQAGBfEjjT7%2FVqqt1RS4yqUVC9DzFIMdS7w4%2FqblnGufz9KLYtWzJAw94ie6aXPLm8efy%2Bp9CW1G5K%2BLnADeKr3CfPgHkJEZT1uE6lmSvF5XRGA%2B9PPVOVLSugosDszousc%2F8Y2I8byI%2FjzRdlq49FZgzdD77cfy7ZOvh8DImi8YcndeDVE%2FouKp47Sc9oSNeSrA%2FOwlJiHE0oMXIJbaHAHimlykfbSsQTFlE36elNM82Mo5rKhmOqAitk6%2BtauwrObS7MhgKo6gKe21A8hZHZDyOw3C1k9qm0Eta9EtBcLJydzLaR12Y%2FTcHjrx5m1%2FLve4DTk77FKW1%2B%2Bp%2FCKJ8NXXTnXvx%2BlMAT%2F9B2nJF3TCFRhr9aUB4z9ENIINhHVJwzRLskkugxEQ5x6g%2Fgw3vWrmhhcqo7Eiam9pYus9r4%2BgHvgs0zu7a6mngW10lhtECR1tSmnhuIISMqA7C8a0Mcw1DW7qKsNw5gb2clN9p2WVvY6Jvum1pwsliLXq%2FUzjZElzebySso080ptRa3z5O9FdsaR2GshHXOHdgEFum325iEG6%2FXp%2FhNwBxoXhsjWYYZ1B6IbHVZJVZXzB5ZiTVJNy2n%2FRFzCkktTaK%2Fkf4X%2BqjOAC7FHlk7zTbO4iQvmdaweagpXNWGfF0zVds7172m1UWnj3nZpGM6TdJ248WLCTks1OdVXRnZabapYrhFIXW1m5gxFLAgaFsg3ZNCvCkhN4YBUxwVSH9tRNfyU%2BoqfGt8ltU2p6127pMvURUzoSaz%2BisL1s%2FWcZhXjiFBcJ%2BwfdG3Z0EbZitewkxpzZG1UP0bmocxdTiRyNLkKFb6UmQ8cDedc%2F54CR62NCfkTrmnY2NqG0O4BxpXNWcd83wkaFc7DS6pbhjUpk17fPFqmY3Ilt0zL9ximCyxK29XJoWpcuOVK1eIE6MRyZWcJDmvCYnW5GPTNXpVbx6yK5AZib%2Fk7a9Y%2B3XmAwPlLYeEnYIl5FVmObYrFpYWvT0%2BUcOP7MfQBK%2FTicaU%2FlTSBASHy2WYfN9sVRAkMEOSC8Ka4sUWel6sbTNAvsM6GYppRhCF0XON2ZtyzsaiGJbmydYQ5V2xrquKqsapIQNf6At0SE17HWobHGO8Pk8ecX6qXgS4T9P4wF9NcLp0tih6jqhR5xRiAZJfvbtygPfMf3ewr4syDJFKTbSzqbXOWJea0rhA1zjFpEqM6LGpiauP6UKtQKt25JBE9LGpiVVIEXygh3YUYePQiwgRtqIljW0vFGq%2BpmUjNamZ6HBF%2FVbqhtTcG2AIDYrih0AQM6GzlBe2ybb0slvxfChPyAZjAWVxDwgXphtjeuCBGZBtIXMYDljJVaAyfAa88Iw%2Fij6cG1ebaOgdkhqg3DthieFZyYBeCA4wXf1HYlX9QoGPgMla4WenNPyfPC53bTllpR50X1qC8ECd91PBk30znmj8yHfiBog2ePPZLbmpY7gkcUQ80AXtJvjRYaGxTzOp5Z5dGacY%2BqW36VBgwITEE25lGP1ohQbpdRwCFScvs69S4bnIFGrKdh1XB%2BCBhmC1OlNIExr8lXBAwfc8jbK2XVBBIse9C4%2BQnioy98m1b3GLbu9ba8h8h7KJ5T3LTbruR07bkvO58MVvKuOpEoWEZ9%2B6DMzrhjdWWNZ3sjnpz6RtXaGXojRrLV%2FtbQ%2Bz0s2U1Q4JH%2FD2ibhXh9KQTFIvl8iTiZBcsVW7F0hx7ydKWbZkSYEevMepkAuiN29IkZxx1gLrGJcolGYmyLKAOem%2Fb1GxxKiyLFyGIs8kZjSHSkJmEu%2FZHWAQgirLj8dixcW96lM4I310jMLlIyeFto76QxEqyRRSjL%2BFwZLOTE6YNbbMzBeuA8%2Fhmh95V1M9gDcNHimJxil15bN5tyG7cVieRlDo40%2FRN9o8ZAfaymzJjq8jSt8X33AeEsEMTb7L9qyvXi5QFcnG0yY5IWbj0jVSOAAH0P0an8rlapwmKYJLMtyCig2wpU%2BfqnMaNK42dpLZa01hkrmr2Yhf5ckORvZyGG9kpLB1aD5XbjlPtKq4nlHVRQkpa99H0KPUAVxJNy8NUMWKT9rs0YLsslh6lOkgeS1tjBtOnWTh4LM3PlA31fCzN9beVbmPpc8J9Plj4Cj0PkfbxQYgYJ6%2BsFvCyE0C50NqS5Hj7Cq2lfHvD%2FpO%2F%2FXg7P36ZgHB%2B3C5LOcby4%2BIcOS60uPfp1EVActMpS3aoXE%2FTqVa2kIkn%2B7yq9KhcUWxpHLOF4CRd59y44gz9hVzk1WHADL08rhUVwkMJhSVDRlMew9RH15Gfvwh5bmnekRSolMdxXZqep83jse95QHI8O1%2F%2F9H8%3D)
   4
   5 Note: this is a work in progress, still unreleased software. Feel free to look around, but keep in mind it's not finished yet.
   6
   7 **Contributors wanted!** Please send me an email (see commit log) or contact `girst` on irc.freenode.net (hanging out in `#invidious` for now)
   8
   9 Test Instance: [https://subscriptions.gir.st/](https://subscriptions.gir.st/)
  10
  11 TODO:
  12  - task queue for refesh-cipher, pull-subs, update-websub
  13     - should handle delays itself
  14     - would allow us to just put a websub/pullsub request into the queue
  15       whenever a user subscribes to a channel, without worrying about
  16       ratelimiting ourselves.
  17  - abstract database access
  18    we want to be able to choose between at least sqlite and postgres
  19  - implement 'dangerous' functions as disablable blueprint
  20    - search through invidious-api
  21    - proxy hls/dash manifests
  22      - hls.js for livestreams
  23  - document all the things
  24  - quality of life improvements:
  25    - asynchronically call update-subs and pull-subs on subscribing, iff necessary
  26  - development-friendly error handling:
  27    - log unkown info/endcards to file
  28  - proper support for sponsorblock (cached db?)
  29    (blocked on https://github.com/ajayyy/SponsorBlockServer/pull/86)
  30  - cleanups:
  31    - move subsystem-specific stuff out of common/common.py (e.g. reddit)
  32    - remove jsonify()
  33    - purge magic strings (e.g guest token)
  34    - fix all the TODOs and XXXs
  35    - clean up login stuff
  36      https://flask-login.readthedocs.io/en/latest/
  37      https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-v-user-logins
  38    - jinja2: for/else
  39    - welcome message: maybe make dismissable, show on all pages?
  40  - find a name!
  41  - /channel/UE.../videos 404s
  42  - invidious search: api returns 'nice' author strings (e.g. numberphile instead of UC...)
  43    if not channel_id.match(...): redirect('/user/...') (in invidious blueprint)
  44  - subscription button not in invidious blueprint; should also work for playlists
  45  - sometimes we cache broken video-urls!!
  46  - on websub: check if livestream and has postlivedvr enabled, if not delete afterwards (need test case!)
  47  - reddit pagination is broken
  48  - info/endcards: when deduplicating websites, ignore url protocol
  49  - reddit manager: buttons not styled correctly
  50  - /watch endpoint should allow continue playing playlist (&list=PL...&index=x)
  51  - test fetch_ajax on blocked instance
  52  - this channel is very suceptible to errors: UCJeTqOt4xDuL2imX7zYnJlw
  53    UC6Om9kAkl32dWlDSNlDS9Iw
  54  - remove all instances of `grep -r macros.card app/`
  55
  56  - NEW CHANNEL PROTOBUF FORMAT https://git.gir.st/subscriptionfeed.git/commitdiff/4e3b56b967570b9498651cf758a5ca3c56640689
  57    this url should return the first video of page 2, which should be from ~1month ago. it works on my local machine, but not on my vps. i suspect, youtube are in the process of migrating schemas and have rolled them out to some regions only.
  58 ```
  59 curl -v 'https://www.youtube.com/browse_ajax?continuation=4qmFsgJoEhhVQ1h1cVNCbEhBRTZYdy15ZUpBMFR1bncaTEVnWjJhV1JsYjNNWUF5QUFNQUU0QWVvREprTm9iMGxuVEdwMk0wMTVlakZqVWxoRlp6UkxRVUpKUzBOT1lWZHFkR2xFT1hCMmRGZDM%253D&hl=en&gl=US' -H 'x-youtube-client-name: 1' -H 'x-youtube-client-version: 2.20200813.04.02' --compressed|jq '.[1].response.continuationContents.gridContinuation.items[0].gridVideoRenderer|[.title, .publishedTimeText]'
  60 ```
  61
  62 stuff to look at:
  63  - app.config.from_pyfile('the-config.cfg')
  64    NOTE: this will not work for the non-python/non-flask parts
  65  - playlist types:
  66    PL<30chars>  normal playlist
  67    PLAC5BDC262E18B063 : normal playlist, older format?
  68    RD<videoID>  "Mix" playlist
  69    RD<24chars>  "Mix" playlist for musicians (e.g. foo fighters == RDEMx2SPzeaRXiOzpOe0SxPVJA)
  70
  71 wishlist:
  72  - proxy googlevideo (and probably thumnails) responses
  73  - LibreJS (SPDX not supported; LibreJS doesn't define a AGPLv3-only identifier)
  74
  75 # Installation
  76
  77 see INSTALL file
  78
  79 # Non-Features
  80
  81  - no algorithmically determined video recommendations: Find additional content through Youtube search, endcards or Reddit
  82  - no youtube comments
  83
  84 # Notes
  85  - caching external api requests
  86    currently using in-memory-backend, which is purged every 10minutes. a real installation should use redis, which should scale better and handle purging for us.
  87    the in-memory cache conflicts with gunicorn's worker model: which is a sepreate process for each request. switch to mulitple gthreads on a single worker to avoid getting the cache torn down after every request (note that threads may leak memory, if my observations from $dayjob are still valid).
  88  - neater sqlite output:
  89     .mode column
  90     .headers on
  91  - eventlet worker on ipv4-only host:
  92    for some reason, using eventlet workers makes requests (which uses urllib3 internally) try to use ipv6 dns lookups, before re-attemtping ipv4, resulting in a long delay before responses. either one of these workarounds is ok:
  93    - put this in `/etc/sysctl.conf` (test with `sysctl -w KEY=VALUE`)
  94         net.ipv6.conf.all.disable_ipv6 = 1
  95         net.ipv6.conf.default.disable_ipv6 = 1
  96    - put this in `app/__init__.py`:
  97         import socket, requests
  98         requests.packages.urllib3.util.connection.allowed_gai_family = lambda: socket.AF_INET
  99
 100 # Advanced Topics
 101
 102 ## Extending Unnamed Youtube Frontend
 103
 104 UYtF is trivially extensible using [Flask Blueprints]. Just drop a new
 105 blueprint in `app/` and register it in the `[frontend]modules` section of
 106 `config.ini`. The Blueprint inside a module is expected to be named `frontend`.
 107 An `example_blueprint` is provided, which shows off how to use `fallback_route`
 108 and fetching the user token on guest-accessible routes.
 109
 110 The default `youtube` blueprint provides unblockable but minimal versions of
 111 the /channel and /playlist endpoints. A custom blueprint (or, as an example,
 112 the `invidious` blueprint) can overwrite existing routes by being listed before
 113 the one to be overridden in `config.ini`.
 114 A blueprint may also delegate a request to another (lower precedence) blueprint
 115 using `return fallback_route(*args, **kwargs)`. This can be useful if the
 116 endpoint might not be as reliable (e.g. invidious-api being blocked)
 117
 118 [Flask Blueprints]: https://flask.palletsprojects.com/en/1.1.x/blueprints/
 119
 120 ## Proxying videos
 121
 122 This is useful to watch videos that are IP-locked. add the `proxy` blueprint to
 123 the `[frontend]modules` list in `config.ini` to enable (disabled by default, as
 124 it is resource-intensive, blocks a gunicorn thread and hasn't been tested
 125 w.r.t. googlebanning).
 126
 127 ## Guest User
 128
 129 The guest user doesn't really exist; you can't log in as guest, and all guest
 130 views are read-only. To modify the shown subscription feed, you'll have to
 131 modify the database manually. To add a single channel, issuing
 132 `INSERT INTO subscriptions (user, channel_id) VALUES ("guest", "UC...")`. To
 133 load many at once, prepare a file (guest.csv) of channel ids
 134 that looks like this:
 135
 136     UCxxxxxxxxxxxxxxxxxxxxxxxx,guest,channel
 137
 138 Then from the `sqlite3` console, issue:
 139     .mode csv
 140     .import guest.csv subscriptions
 141
 142 ## Running modules on different Hosts
 143
 144 The `webhooks` and `proxy` modules support being ran standalone on a different
 145 machine. For this, remove them from `config.ini`'s `[frontend]modules` and
 146 start them using gunicorn by replacing `app:app` with (e.g.)
 147 `app.webhooks:app()` (the parenthesis are required, as those use the factory
 148 pattern to not uselessly instantiate the Flask app object if it wouldn't be
 149 needed). An example is given in `config/subscriptions-webhooks.service`.
 150
 151 Note that the proxy endpoint can't reliably access ip-restricted streams if its
 152 IP is different to the frontend's.
 153
 154 ## License
 155
 156 This project is licensed under the [GNU Affero General Public License, Version 3].
 157
 158 [GNU Affero General Public License, Version 3]: LICENSE