Feat: Implement pagination for playlists when using Invidious API

[Shadorc]

Pull Request Type

• Bugfix
• Feature Implementation
• Documentation
• Other

Related issue

closes #2333
This is considered to be a bug, but the fix needs implementing a feature.

Description

Add support for Invidious playlists pagination.
Also, fix the video count in the playlist description to display the total count.

There are three ways to implement pagination with Invidious API, two of them are undocumented:

• page: Inconsistent behaviour (see [Bug] Playlists API endpoint 'page' param is inconsistent iv-org/invidious#5816) which makes it hard to rely on for caching
• index: More consistent than page, it is the video index in the playlist. A lookback window of 50 is applied to the value
• continuation: Same as index but with a video_id. It is commented as being expensive.

I chose index, which is the less expensive and the more consistent. However, I had to compensate for the lookback window which is not useful in our case (I don't know in which scenario it could be useful too...).

Testing

All tests require Settings > General > Preferred API Backend to be set to Invidious API with an Invidious instance having its API enabled.

Large playlist can be fully loaded

1. Go a playlist with 200+ videos (e.g. https://youtube.com/playlist?list=PLj3LVjI7cF7k-LmGKRiw6XwbU8SAtQwCF)
2. Check that the playlist description displays the total video count
3. Scroll to the bottom
4. Check that the Load More Videos button appears
5. Click and check that more videos are loaded
6. Repeat until reaching the end of the playlist

Playlist cache is conserved when navigating to watch page
This one requires some feedback from the code. I would suggest adding a log to the invidious#invidiousGetPlaylistInfo function.

1. Complete all the steps in the previous test
2. Click on a video in the playlist
3. Check that invidiousGetPlaylistInfo is not called
4. Check that the playlist view on the watch page contains all the videos

Watch page loads the full playlist

1. Go a playlist with 200+ videos (e.g. https://youtube.com/playlist?list=PLj3LVjI7cF7k-LmGKRiw6XwbU8SAtQwCF)
2. Click on any video
3. Check that the playlist view on the watch page contains all the videos

Direct acces to watch page loads the full playlist

1. Navigate directly to the watch page of a video within a playlist (e.g. https://youtu.be/h5cDiLHs2JM?list=PL47cI9uL16XuIfObjdqaP6HFdyjYcEVyT)
2. Check that the playlist view on the watch page contains all the videos

Regression
Check that the behaviour hasn't changed for:

• small playlist using Invididous API
• small and large playlists using Local API
• user playlists

Desktop

• OS: Bazzite
• OS Version:
• FreeTube version: v0.25.1-beta

[PikachuEXE]

Can't find a working instance to test

[absidue]

Just like PikachuEXE and all other FreeTube maintainers, I have no way to test this pull request, as there are no public Invidious instances that have the API enabled (going through all the effort to set up an Invidious server on a Windows machine just to test this one pull request is not worth it).

You are also missing a few test cases:

1. Navigating to the watch page after loading all videos on the playlist page should carry over all videos.
2. Open the playlist page and without loading anymore videos on the playlist page, navigate to the watch page, the watch page should carry over the already loaded videos and load the remaining playlist videos.
3. Navigate directly to the watch page with a playlist (/watch URL with the &list= query parameter), the watch page should retrieve the entire playlist.

However, I had to compensate for the lookback window which is not useful in our case (I don't know in which scenario it could be useful too...).

It is useful if you want to show some videos before and after the current video on the watch page (that is probably also why there is an undocumented index parameter in the Invidious API). FreeTube loads the entire playlist on the watch page. The look back window combined with Invidious trying to return 2 YouTube pages (100 videos) worth of videos for every Invidious page (200 videos), are what are causing the problems.

[Shadorc]

Indeed, when I opened this MR, I was worried that it could be hard to test, given that no public Invidious instances have their API enabled.
I have set up my own instance to implement this, and I was wondering if I could open it for you, with authentication, but I don't leave my PC on 24/7, so it would be a bit messy for you to test.
I was going to search for some unlisted public instances with an API, but I don't really have any hope of finding any.
It could be decided that Invidious feature implementation is frozen while no public instances are available, but I fear that it could last forever. Anyway, I would be fine with this decision.

Thanks for the additional test cases!
I have already tested them while implementing this, but I forgot to write them in the MR.

Edit: Test cases updated

[radmorecameron]

My computer is set up for Invidious development, I can probably test this in ~12 hours

[radmorecameron]

[x] Large playlist can be fully loaded
[x] Playlist cache is conserved when navigating to watch page
[x] Watch page loads the full playlist
[x] Watch page loads the full playlist
[x] regression: small playlist using Invididous API

Seemed to work well with your example playlist. I tested with Invidious api + fallback api set to off.

But, with this playlist: https://youtube.com/playlist?list=PLbio2sNGiPZOu6C1XtpLKVHHsheIP2MsH

I got some weirdness, the load more videos appears after 410 instead of 400? I'm guessing this is just due to removed videos though. edit: happens through invidious web interface too, it's because some videos were removed

[radmorecameron]

Works great & nothing about the code is concerning me, Note: I only tested with Invidious (not local api nor user playlist)

[Shadorc]

Thanks for taking the time to test!

Indeed, Invidious returns deleted videos but does not count them, so it can return > 200 videos per batch

[efb4f5ff-1298-471a-8973-3d47447115dc]

All cases and more stuff i could think of passed testing. Tested on Pika's private instance

[Shadorc]

Has anyone tested with a playlist containing a large number of videos that are unavailable?
I've tried with https://youtube.com/playlist?list=PLvMVt4c68EeXRV2R1fdieei9HwpuUKXKL because I was checking if the load more button was still appearing even if the playlist was complete (without all unavailable videos).
This seems to completely kill my invidious instance.
Lot of thumbnails are not loading, the load more button is here and clicking it does nothing, and after that, every requests to the API never returns.
It feels like the instance is stuck in a loop, or maybe I've just setup my instance badly, but it works with the playlist used as an example in this MR.

Update 1: I have the same behaviour on development so it's not related to this MR
Update 2: That's weird, because when the instance is not reachable anymore in FreeTube, I'm still able to access the API via my browser, and it responds
Update 3: It's an issue with thumbnails for unavailable videos. If I set Thumbnail Preference setting to Hidden, I'm able to fully load the playlist. This is good for this MR then, Invidious returns all the videos, even unavailable ones.

[github-actions]

This PR is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.