I have had the same issue on multiple sites on D10 - sites on D8 are not experiencing this.
The issue only happens when logged in as Admin, when visiting the site as a regular user im able to load all the content from the load more button in the infinite view.
using Views Infinite Scroll module 2.0.2
I dont have any sites on 9 to test but assuming this is a D9 and up issue?