README.md

   1 # Unnamed Youtube Frontend
   2
   3 [rough functional overview diagram](https://app.diagrams.net/?lightbox=1&highlight=0000ff&edit=_blank&layers=1&nav=1&title=youtubefrontend6.drawio#R5Vxbc6M4Fv41rtl5cIo7%2BDGXds9U9dSmJg%2BzvS9dAmRQt4xYEIk9v34lLjZIckw6BkwmL0EHAeY737noSGJh3m93nzOQxn%2BQEOKFoYW7hfmwMAzd0j32j0v2lcTTrEoQZSisOx0FT%2BhvWAu1WlqgEOadjpQQTFHaFQYkSWBAOzKQZeSl221DcPepKYigJHgKAJalf6GQxvVbGO5R%2FhtEUdw8WXdW1ZktaDrXb5LHICQvLZH5aWHeZ4TQ6mi7u4eYg9fgUl23PnH28MMymNA%2BF2D76fOGuN%2B%2B%2Bl9X%2F%2F12%2F8eTHm2WtlPd5hngon7j%2BtfSfQNBRookhPwu2sK8e4kRhU8pCPjZF6Z0JovpFrOWzg43CON7gknG2glJWKe7EORxeTk%2FXz8OZhTuTr6IfoCH8QqSLaTZnnXZdalRM8qpmy9H9RhmLYtbqrGtWghqSkSHOx9RYwc1cG8AsblxCzMYMhbVTZLRmEQkAfjTUXrXRfXY5wshaY3Vd0jpvjYJUFDSRZqhle3%2FU19fNr7yxo3dNB927ZMP%2B7qVU5DRW24aRxWVsjXir11fETY9AgzyHAWVsO5yVCR%2Fz9fVyGAhRRbAV%2FBr7BpkEaSv9bPVvMggBhQ9d3%2BISsvlpezFwL7VISUooXnrzo9c0KKb1%2BWbXdv%2BkTHVHY%2F8Ofy0n6eULpklJswpcUcKKPBBDiXKMf%2BS8kPeIackg%2BeN9RLGaHfRMT3ZHB2FNTpDGaO5mtYYb2y7a4%2BrcwZZth5hhhgAMLtaKzV7WukkRmo2YbehoSkEQ6G%2F49qv9R%2FGqE1FrHUw5SESPbPDiB%2FePv7eSNlDWicUfQMQxPBU7xPuIdhjxLienfcOfmUUX%2FyDAAQ%2FotJU%2Fl1Qdhd4OTdimoL%2BpnYjtjWFG7k%2Bq7fGis3vy8DOZ7EN%2F0FASQ%2FyX4DTjtUNjdbKlThtjclpZ86chjtEqwBr1a0qvDbnjsGVN%2FathhhaL2gddl%2FrMKYIio7ldAio6yMEOVf%2FECTzOiTzJifZe7kjk8MRvZPgdipS11cJnucCPLEll%2F2VFLTwWT6j8YcZ2mdCIgxZPpTLjMIYpXmp5CatwaQIR3HrttsFTjfkVEVfKfy6PlgBwu0f%2FmK4A8y0GBRpi7G19EBiYxQkTd3oIGlrKwlJFZDDBUhzCt%2FVGjuah3bL78x%2F6LjqGSYnGTo6Rnfo0VBu0Cjp2BMzzROqFNoHqVL0pdoJzzTOcGWl8NfS4P4epTEDum8t4HMGfP90%2FxmEg0PVvjFERWV%2F3HAgh9FxjVTXhXjgau7HMNNmDuc6Q4IlTDEZZ6qJUn99NUII8SZmp9MNIf8wYr43gPwUMQ2tW7Y2vNeJqXur1%2FoPNHel9QlvaYHxMi%2F8PMhuZhy0dGEYbWgTBy1dNcswAyBNvVusshuqTwbkG4rKVwWkoQlA6tMCaU4dqDpxSj8Tp64wJPWtMpvvDEnvMxe5oKfw%2BmsG14%2FeY5p1RhLKcJ1xfDANq2ONqkmgcQc1zsTWqH%2FQykNTzLzOzNER1juIiwXFxNFevdZ%2FoMTR6ONC%2FoI%2BSxt7%2BxDWPSbk4HP8TOw7Q6ciRHjFWolxXcqsp%2FyOnulYXKk8k%2FUzjumSHqXvwshhZgjtVZdmuisQaOAZQsedMa0uToOLq1cX5jEPS65GUm%2FD7pa3L9IQULh8qT38DDyxaCLW1K5Ynlr48%2BmJCW6jKIMR4GufRFzZ29MuMACjKOHkZiiUKwU5RigA%2BLY%2BsUVhWJkbzNHfwC9vxS2jTkPYfe27hf3A78UsLK%2BM7UKYC0s3HV2eNdY1BejGUKC7csHruJbhMSO7%2FewxF6fqVaCriD4c5nKZK2CjRTliHFaKnHEGIE%2Br%2FUMbtOPx4zI7VQT3oMjUVFtVzMFQk2taV4iaEJgMhVMdFzW5tHF9qBnCalNTUYgeF7UeW8uakM5ekyKA%2F2SogCTCPSD0CaVkWzey%2BnX4MYab5pCWKR93kYpdaSxRpCiJ7kt3y%2FP%2FZVUo7OxWywhlKRbPNR6WK%2B0yivKEKRxdrnPbqmXBA2pKXj%2BWwBcmKFJMQMgOEkLRhgUjjoW8Gm9uwcxwuzW5lWwpulIDgynAkxSQwQ2DJuZAl3NkKC15yLP%2B%2FxUwpx9ACUJstBVaUG35GE4Lcu68gTTgOuA2qrHRVgm89oxCSD6eGRw21rY1oAoZg2nAkxPpRgMpBnuY3XxnsGv%2FYkBnIOCqCMpFUr%2FOXhemsI3yYB1tXbij6kIenjPHU%2F7mCNJvpQ18Q8mGzB57SxjEN3sBJ7QDQ5E6SZXt%2BMbgO3rWz77XOKVSERkE24Vx37tCTuNi6ycA4bxnSXxuCha3Oqg2FyoD%2FmAZlyePXoscZr%2FkQryff5CRds8rlnUcdlCMY1zTTIA2O4sW7X1F9uK1fUXXV4r2LLWy31mKNiWS9CtFv3mXuPAcyxlhVtObevmL5woz7h9kwt3ruzfZcy%2FB2reSzRYG%2BuYY%2B0o81aI1KeL%2FnrBsAZHiZMCXV2tWVezZzpjb4l5sRY436kSNp9qzJ8GOzinqZAXyjfsiFQWyS6Au1L9NxaimWZjSRn2wjZKePMRXoJ5DkJWDTpYwFZjbeP9kOohBkpTfQOOfBnvTpWyk%2B9OrVOaWFlrmeXtUTuLZQ1FjpRpznfBsaFt%2BNa4Nt4huWXs%2BSL8AH%2BJHhmFdTG6q13eYn7g7fFelMcGFYW7KP%2B4D%2BMNumxkDriBp%2BqD%2BPQ8xpfyzeLfl%2Ful1ECbaDQpIsik%2F%2BHITsCcyFgEK2D8uZ%2Bxc%2B0WOEpjnyy1I2E22TH9LfclSlrXBvw239peGttQN7yZNIrWfKB%2FOUpTymzKXdB5CScRyDYkipikzpJG9gSGsefwOXxWKj18zND%2F9Hw%3D%3D)
   4
   5 Note: this is a work in progress, still unreleased software. Feel free to look around, but keep in mind that this repository will move servers for its public release.
   6
   7 Contributors wanted! Please send me an email (see commit log) or contact `girst` on irc.freenode.net (hanging out in `#invidious` for now)
   8
   9 Test Instance: http://delta.gir.st:8000/ (will move in the future)
  10
  11 TODO:
  12  - task queue for refesh-cipher, pull-subs, update-websub
  13     - should handle delays itself
  14     - would allow us to just put a websub/pullsub request into the queue
  15       whenever a user subscribes to a channel, without worrying about
  16       ratelimiting ourselves.
  17  - webhook: as blueprint; main app looks in envvar/config what to load?
  18  - abstract database access
  19    we want to be able to choose between at least sqlite and postgres
  20  - implement 'dangerous' functions as disablable blueprint
  21    - search through invidious-api
  22    - proxy for subtitles, hls/dash manifests, ...
  23      - subtitles on /watch
  24      - hls.js for livestreams
  25    - proxy for geolocked videos?
  26  - document all the things
  27  - quality of life improvements:
  28    - asynchronically call update-subs and pull-subs on subscribing, iff necessary
  29    - get_video_info on incoming websub-webhook to get exact published date
  30  - development-friendly error handling:
  31    - log unkown info/endcards to file
  32    - save get_video_info/reddit.json/etc with traceback
  33  - proper support for sponsorblock (cached db?)
  34  - allow pinning non-subscribed videos
  35    would make reddit easier, less-surprising when unsubscribing from channels with pinned videos
  36  - cleanups:
  37    - move subsystem-specific stuff out of common/common.py (e.g. reddit)
  38    - remove jsonify()
  39    - purge magic strings (e.g guest token)
  40    - fix all the TODOs and XXXs
  41    - clean up login stuff
  42      https://flask-login.readthedocs.io/en/latest/
  43      https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-v-user-logins
  44    - jinja2: for/else
  45    - welcome message: maybe make dismissable, show on all pages?
  46  - find a name!
  47  - /channel/UE.../videos 404s
  48  - invidious search: api returns 'nice' author strings (e.g. numberphile instead of UC...)
  49    if not channel_id.match(...): redirect('/user/...') (in invidious blueprint)
  50
  51 stuff to look at:
  52  - app.config.from_pyfile('the-config.cfg')
  53    NOTE: this will not work for the non-python/non-flask parts
  54
  55 wishlist:
  56  - proxy googlevideo (and probably thumnails) responses
  57
  58 # Installation
  59
  60 see INSTALL file
  61
  62 # Usage
  63
  64 Since user authentication is not yet implemented, users are for now identified by a randomized token. To "log in", just visit `/feed/subscriptions?token=....`. at the bottom, click 'Manage Subscriptions' and paste the UC...-URLs of channels you want to subscribe to.
  65
  66 If no token is specified, it defaults to `guest`. Since this is a readonly account, you can populate it by directly writing into the database:
  67
  68 first, prepare a file (guest.csv) of channel ids like so:
  69     UCxxxxxxxxxxxxxxxxxxxxxxxx,guest,channel
  70 then from the sqlite3 console, issue:
  71     .mode csv
  72     .import guest.csv subscriptions
  73
  74
  75 # Notes
  76  - caching external api requests
  77    currently using in-memory-backend, which is purged every 10minutes. a real installation should use redis, which should scale better and handle purging for us.
  78    the in-memory cache conflicts with gunicorn's worker model: which is a sepreate process for each request. switch to mulitple gthreads on a single worker to avoid getting the cache torn down after every request (note that threads may leak memory, if my observations from $dayjob are still valid).
  79  - neater sqlite output:
  80     .mode column
  81     .headers on
  82
  83 # Advanced Topics
  84
  85 ## Extending Unnamed Youtube Frontend
  86
  87 UYtF is trivially extensible using [Flask Blueprints]. Just drop a new
  88 blueprint in `app/` and register it in the `[frontend]modules` section of
  89 `config.ini`. The Blueprint inside a module is expected to be named `frontend`.
  90
  91 The default `youtube` blueprint provides unblockable but minimal versions of
  92 the /channel and /playlist endpoints. A custom blueprint (or, as an example,
  93 the `invidious` blueprint) can overwrite existing routes by being listed before
  94 the one to be overridden in `config.ini`.
  95 A blueprint may also delegate a request to another (lower precedence) blueprint
  96 using `return fallback_route(*args, **kwargs)`. This can be useful if the
  97 endpoint might not be as reliable (e.g. invidious-api being blocked)
  98
  99 [Flask Blueprints]: https://flask.palletsprojects.com/en/1.1.x/blueprints/