README.md

   1 # Unnamed Youtube Frontend
   2
   3 [rough functional overview diagram](https://viewer.diagrams.net/?title=youtube-frontend-2020jul19#R7Vxbk5s4Fv41rtl9sIuLuT32Je6d2kylK121k%2BxLSgYZNMHIC6Lbzq9fiZtBkt10m4vpmjx00EEI%2FJ2rjo400%2B%2B2%2B4cY7II%2FsAfDmaZ4%2B5l%2BP9M0R1XoX0Y45ARbWeYEP0ZeTlKPhCf0CxbE4jk%2FRR5MGh0JxiFBuybRxVEEXdKggTjGL81uGxw237oDPhQITy4IReqfyCNB8Ss060j%2FF0R%2BUL5ZNZ38zhaUnYtfkgTAwy81kv5ppt%2FFGJP8aru%2FgyHDrsQlf2514m71YTGMSJsHQuPpYYOtH9%2FX353%2F%2Frj740n1N3PLzId5BmFa%2FOLia8mhhIB%2B%2BI5d0lcRBMKvFGYQ%2Bezu7UuACHzaAZfdf6ECQGkB2Ya0pdLLNSYEb4tGXKDErkO4KS8Jpg%2FdK%2FRqg8LwDoc4pu0IR2z8GKcERf4d%2FYkw%2Fkbpc2VhULoHkgB65cCYAIJwxG47bCARmQKsZxgTuK%2BRCqQeIN5CEh9ol%2BKubeRPFFKrqUX75SgDxrJgbFDjv6YXRFDInV8NfWQNvSi4I%2BfU%2Fb9%2FqU%2F48ctPLfmy0a3%2FfJrHaK7qr3OKwhV5GS7K68yRwN2A9QRkEmBPolgqcYGiaSoCikuzLxSl8m60kPceULxIFnkURRArvOogVvLZOYjlwDXMoEetZtHEMQmwjyMQfjpSb5uoHvt8xpkBYFj9BQk5FC4ApAQ3kU4ItT83zKQfoc5oK8Q%2BPxsWRl7Zww1BkiA3JxZd2DBwj8i3sju9%2Fs6uM6PCWvf72q37Q9E4ycQEp7ELz0BV2g36nT4k5zrm%2FRiOZ2UihiE1ds9N7yTjcPYoBQMcah12GEUkqY38yAg1UbObsmYUfu4oLfmIR9mpPu394lQqxFElQ0wdMAsaAAFrkMCTLol1SAiOW%2FiiLhTRaKKj26IqysyZ2Zci6s4YikgBjA%2FfcrUxynauRk7VPipS1jrUW48wRhQAGBfEjjT7%2FVqqt1RS4yqUVC9DzFIMdS7w4%2FqblnGufz9KLYtWzJAw94ie6aXPLm8efy%2Bp9CW1G5K%2BLnADeKr3CfPgHkJEZT1uE6lmSvF5XRGA%2B9PPVOVLSugosDszousc%2F8Y2I8byI%2FjzRdlq49FZgzdD77cfy7ZOvh8DImi8YcndeDVE%2FouKp47Sc9oSNeSrA%2FOwlJiHE0oMXIJbaHAHimlykfbSsQTFlE36elNM82Mo5rKhmOqAitk6%2BtauwrObS7MhgKo6gKe21A8hZHZDyOw3C1k9qm0Eta9EtBcLJydzLaR12Y%2FTcHjrx5m1%2FLve4DTk77FKW1%2B%2Bp%2FCKJ8NXXTnXvx%2BlMAT%2F9B2nJF3TCFRhr9aUB4z9ENIINhHVJwzRLskkugxEQ5x6g%2Fgw3vWrmhhcqo7Eiam9pYus9r4%2BgHvgs0zu7a6mngW10lhtECR1tSmnhuIISMqA7C8a0Mcw1DW7qKsNw5gb2clN9p2WVvY6Jvum1pwsliLXq%2FUzjZElzebySso080ptRa3z5O9FdsaR2GshHXOHdgEFum325iEG6%2FXp%2FhNwBxoXhsjWYYZ1B6IbHVZJVZXzB5ZiTVJNy2n%2FRFzCkktTaK%2Fkf4X%2BqjOAC7FHlk7zTbO4iQvmdaweagpXNWGfF0zVds7172m1UWnj3nZpGM6TdJ248WLCTks1OdVXRnZabapYrhFIXW1m5gxFLAgaFsg3ZNCvCkhN4YBUxwVSH9tRNfyU%2BoqfGt8ltU2p6127pMvURUzoSaz%2BisL1s%2FWcZhXjiFBcJ%2BwfdG3Z0EbZitewkxpzZG1UP0bmocxdTiRyNLkKFb6UmQ8cDedc%2F54CR62NCfkTrmnY2NqG0O4BxpXNWcd83wkaFc7DS6pbhjUpk17fPFqmY3Ilt0zL9ximCyxK29XJoWpcuOVK1eIE6MRyZWcJDmvCYnW5GPTNXpVbx6yK5AZib%2Fk7a9Y%2B3XmAwPlLYeEnYIl5FVmObYrFpYWvT0%2BUcOP7MfQBK%2FTicaU%2FlTSBASHy2WYfN9sVRAkMEOSC8Ka4sUWel6sbTNAvsM6GYppRhCF0XON2ZtyzsaiGJbmydYQ5V2xrquKqsapIQNf6At0SE17HWobHGO8Pk8ecX6qXgS4T9P4wF9NcLp0tih6jqhR5xRiAZJfvbtygPfMf3ewr4syDJFKTbSzqbXOWJea0rhA1zjFpEqM6LGpiauP6UKtQKt25JBE9LGpiVVIEXygh3YUYePQiwgRtqIljW0vFGq%2BpmUjNamZ6HBF%2FVbqhtTcG2AIDYrih0AQM6GzlBe2ybb0slvxfChPyAZjAWVxDwgXphtjeuCBGZBtIXMYDljJVaAyfAa88Iw%2Fij6cG1ebaOgdkhqg3DthieFZyYBeCA4wXf1HYlX9QoGPgMla4WenNPyfPC53bTllpR50X1qC8ECd91PBk30znmj8yHfiBog2ePPZLbmpY7gkcUQ80AXtJvjRYaGxTzOp5Z5dGacY%2BqW36VBgwITEE25lGP1ohQbpdRwCFScvs69S4bnIFGrKdh1XB%2BCBhmC1OlNIExr8lXBAwfc8jbK2XVBBIse9C4%2BQnioy98m1b3GLbu9ba8h8h7KJ5T3LTbruR07bkvO58MVvKuOpEoWEZ9%2B6DMzrhjdWWNZ3sjnpz6RtXaGXojRrLV%2FtbQ%2Bz0s2U1Q4JH%2FD2ibhXh9KQTFIvl8iTiZBcsVW7F0hx7ydKWbZkSYEevMepkAuiN29IkZxx1gLrGJcolGYmyLKAOem%2Fb1GxxKiyLFyGIs8kZjSHSkJmEu%2FZHWAQgirLj8dixcW96lM4I310jMLlIyeFto76QxEqyRRSjL%2BFwZLOTE6YNbbMzBeuA8%2Fhmh95V1M9gDcNHimJxil15bN5tyG7cVieRlDo40%2FRN9o8ZAfaymzJjq8jSt8X33AeEsEMTb7L9qyvXi5QFcnG0yY5IWbj0jVSOAAH0P0an8rlapwmKYJLMtyCig2wpU%2BfqnMaNK42dpLZa01hkrmr2Yhf5ckORvZyGG9kpLB1aD5XbjlPtKq4nlHVRQkpa99H0KPUAVxJNy8NUMWKT9rs0YLsslh6lOkgeS1tjBtOnWTh4LM3PlA31fCzN9beVbmPpc8J9Plj4Cj0PkfbxQYgYJ6%2BsFvCyE0C50NqS5Hj7Cq2lfHvD%2FpO%2F%2FXg7P36ZgHB%2B3C5LOcby4%2BIcOS60uPfp1EVActMpS3aoXE%2FTqVa2kIkn%2B7yq9KhcUWxpHLOF4CRd59y44gz9hVzk1WHADL08rhUVwkMJhSVDRlMew9RH15Gfvwh5bmnekRSolMdxXZqep83jse95QHI8O1%2F%2F9H8%3D)
   4
   5 Note: this is a work in progress, still unreleased software. Feel free to look around, but keep in mind it's not finished yet.
   6
   7 Contributors wanted! Please send me an email (see commit log) or contact `girst` on irc.freenode.net (hanging out in `#invidious` for now)
   8
   9 Test Instance: [https://subscriptions.gir.st/](https://subscriptions.gir.st/)
  10
  11 TODO:
  12  - task queue for refesh-cipher, pull-subs, update-websub
  13     - should handle delays itself
  14     - would allow us to just put a websub/pullsub request into the queue
  15       whenever a user subscribes to a channel, without worrying about
  16       ratelimiting ourselves.
  17  - webhook: as blueprint; main app looks in envvar/config what to load?
  18  - abstract database access
  19    we want to be able to choose between at least sqlite and postgres
  20  - implement 'dangerous' functions as disablable blueprint
  21    - search through invidious-api
  22    - proxy for hls/dash manifests, ...
  23      - hls.js for livestreams
  24    - proxy for geolocked videos?
  25  - document all the things
  26  - quality of life improvements:
  27    - asynchronically call update-subs and pull-subs on subscribing, iff necessary
  28  - development-friendly error handling:
  29    - log unkown info/endcards to file
  30  - proper support for sponsorblock (cached db?)
  31    (blocked on https://github.com/ajayyy/SponsorBlockServer/pull/86)
  32  - cleanups:
  33    - move subsystem-specific stuff out of common/common.py (e.g. reddit)
  34    - remove jsonify()
  35    - purge magic strings (e.g guest token)
  36    - fix all the TODOs and XXXs
  37    - clean up login stuff
  38      https://flask-login.readthedocs.io/en/latest/
  39      https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-v-user-logins
  40    - jinja2: for/else
  41    - welcome message: maybe make dismissable, show on all pages?
  42  - find a name!
  43  - /channel/UE.../videos 404s
  44  - invidious search: api returns 'nice' author strings (e.g. numberphile instead of UC...)
  45    if not channel_id.match(...): redirect('/user/...') (in invidious blueprint)
  46
  47 stuff to look at:
  48  - app.config.from_pyfile('the-config.cfg')
  49    NOTE: this will not work for the non-python/non-flask parts
  50
  51 wishlist:
  52  - proxy googlevideo (and probably thumnails) responses
  53  - LibreJS (SPDX not supported; LibreJS doesn't define a AGPLv3-only identifier)
  54
  55 # Installation
  56
  57 see INSTALL file
  58
  59 # Non-Features
  60
  61  - no algorithmically determined video recommendations: Find additional content through Youtube search, endcards or Reddit
  62  - no youtube comments
  63
  64 # Notes
  65  - caching external api requests
  66    currently using in-memory-backend, which is purged every 10minutes. a real installation should use redis, which should scale better and handle purging for us.
  67    the in-memory cache conflicts with gunicorn's worker model: which is a sepreate process for each request. switch to mulitple gthreads on a single worker to avoid getting the cache torn down after every request (note that threads may leak memory, if my observations from $dayjob are still valid).
  68  - neater sqlite output:
  69     .mode column
  70     .headers on
  71
  72 # Advanced Topics
  73
  74 ## Extending Unnamed Youtube Frontend
  75
  76 UYtF is trivially extensible using [Flask Blueprints]. Just drop a new
  77 blueprint in `app/` and register it in the `[frontend]modules` section of
  78 `config.ini`. The Blueprint inside a module is expected to be named `frontend`.
  79
  80 The default `youtube` blueprint provides unblockable but minimal versions of
  81 the /channel and /playlist endpoints. A custom blueprint (or, as an example,
  82 the `invidious` blueprint) can overwrite existing routes by being listed before
  83 the one to be overridden in `config.ini`.
  84 A blueprint may also delegate a request to another (lower precedence) blueprint
  85 using `return fallback_route(*args, **kwargs)`. This can be useful if the
  86 endpoint might not be as reliable (e.g. invidious-api being blocked)
  87
  88 [Flask Blueprints]: https://flask.palletsprojects.com/en/1.1.x/blueprints/
  89
  90 ## Guest User
  91
  92 The guest user doesn't really exist; you can't log in as guest, and all guest
  93 views are read-only. To modify the shown subscription feed, you'll have to
  94 modify the database manually. To add a single channel, issuing
  95 `INSERT INTO subscriptions (user, channel_id) VALUES ("guest", "UC...")`. To
  96 load many at once, prepare a file (guest.csv) of channel ids
  97 that looks like this:
  98
  99     UCxxxxxxxxxxxxxxxxxxxxxxxx,guest,channel
 100
 101 Then from the `sqlite3` console, issue:
 102     .mode csv
 103     .import guest.csv subscriptions