So sad that infinite scrolling is kinda useseless due to this issue :/ especialy for all kinds of columnized views layouts.
Not that I have a great Idea to fix this.. but I think, however the solution will look like.. it will very likely require all views plugins to add support for this, by adding for eg. a data-attribute on the right wrapper for this specific plugin.
If this really is the case, we should see, if we could bring all infinite scrolling modules together. It doesn't make any sense to me, that we have 3 (?) infinite scrolling modules, but at least they should use the same data-attribute (or whatever it will be).
As a first step, for me as a themer, it would be fine to have a data-attribute, which I can set myself in a template override, to fix unsupported views plugins.