Loading times decrease for directories containing 100+ of medias

I've had this issue cause an error on two of my sites with thousands of media items and the error message is so unhelpful (http 500 error with ajax) it took me hours to realize what was happening the first time. I'd suggest adding pagination to the default view as that fixes it, though slightly less elegantly than the suggested workaround. But at least it doesn't require installation of another module or a dependency on that module from this one.

This issue seems to be coming up again and again. I have it on one site as well. Views is just so slow when trying to render a lot of entities at once. It needs some POC first to see how different it is.

So we should work on it.
I'm proposing two options:
1. Integrate views_infinite_scroll (either as a dependency or switch if the module is installed on the site)
2. Add a second listing type based on the entity query, which is way faster than Views (in both loading entities and rendering)

Thank you all for your inputs, and yes, we definitely should address this now.

Demand seems here, and we should have larger installations in mind.

Why not just introduce pagination, using the current admin themes pagination at the end of the scroll-able media area ?
Although, I don't like the idea of having another dependency ( infinite scroll → , which I like a lot) - especially for something we can do our self with minimal effort.

Thinking about that, an endless scroll would be a great fit IMO.

Why not implement our own views pagination, just having a load more button maybe having one or two settings like pagesize (bonus:auto-scroll when in viewport setting).

If we also could support the regular pager, all are still free to choose and we have the problem of directories (or ROOT) loading slow eliminated our self.

The idea of using direct entity queries would be an option for me, if loading times would increase dramatically ..
Anyway, who really needs to scroll thousands of medias, I always use the search. I can only imagine a special case, where I search for a thumbnail.

The issue is with sort I guess. We could add a pager, but because the sort direction seems to be oldest to newest, we can't see the last uploaded media which is most likely the one the user wants to select. It is an easy change.
As a user, I usually scroll down and select media there. Most of the time the media is not moved to the directory for various reasons.

I could take a quick look at how rendering with entity query could look like and give some performance differences. View performance is sometimes problematic, browsers most likely can handle rendering thousands of items just fine. Using entity query will lose some customizability some users need so we could add a setting for backend selection.

Good point with the sort, did not think about, we should definitively to change to newest first, or even expose the sort with options alphabetical/newest first/oldest first.

As I said, I would not go with an entityquery and stay with views as it offers a lot of customization just via the UI (no code).
But still, if you want, check the difference, we can introduce another setting - entityquery/regular view ;)

Whereas a pager is needed anyway IMO, as making sense, and aligns with core.

Guys, it could be funny to read "Unsure about pagination ... Locally, with 1000 media entities" in pager discussions if it isn't a problem. A view fully loads each entity, and having no pager its only a question of entities count and allowed memory when you'll get a "500: Allowed memory size ..." error.

After last module update the media_directories_base view was overridden in an update. Previously we customized the view with pager and sorts. And after the update we've got "Allowed memory size exhausted" error. We have 512M memory limit (higher than minimal recommended) and ~10 000 media entities only. It isn't much, I worked on projects where were hundreds of thousands media entities.

The pagination is must be implemented. And looks like it is supported from the box - I've just added "full" pager into the view an everything works.

Yeah - this was 2021 with the mindset that we have directories (always showing everything like the file manager of your favorite OS),
and if I would have 100K of medias, I would have a ton of folders and not having all medias shown in root.
But no doubt, we will now add a pagination out of the box to reduce support in this direction.

The question remains, shall it be:

• Regular Drupal pagination
• Custom Pager
• Another module dependency

Whereas I vote for the custom pager.

And we should change sorting to newest first.

I suggest something like this:

1. Change the sort order
2. Add normal full pager (to go to specific page easily) as default
3. Show 50(?) items per page

The next question would be how to make the update - just overwriting things we need or trying to detect custom changes.

@rang501 - the view already was fully overridden recently in media_directories_ui_update_9002(). It isn't good and was a surprise for us, but if it is already done I think it could be repeated.

@dewalt Updating configuration is complicated, we have two problems here -
1. If we update it, it will override customizations.
2. If we don't update it, users must do it manually which is probably more confusing.

It is hard to find the right balance and detecting if we need to update something is not reliable. We might be able to check if some pager or sort is already added and skip the update, but for more complex changes it is just too difficult to keep track of things.

I would not introduce an update here.

This is for new installations, already running ones will have made their customization, eventually ;)

I added now the full pager (with 18 items so we get 3 rows on desktops) and the sort (by changed, ascending).

@ytsurk
I think the page size should be larger and should reflect the maximum upload size limit, otherwise, we will have a situation, where the user uploads like 20 files, and some of them are on different pages. I did test it and after selecting 20 files, the field had only 18 added.

The default PHP limit for files is 20 and I think it's something that's changed less frequently than upload size.

Good point - I really liked the fact of having three rows w/o scrolling :D

Why not make a warning on the status page, when you can upload more files then we display, with a recommendation to change the views pager size manually?

I want to check how core media handles this case. As I already committed the changes to the -dev branch we need to hurry up here ...