Alter Content View Mode programatically?

I've got a use case I'm working on where I want the results of a view (displaying content) to display differently if there's one result, or if there's many. The way I considered doing this was by altering the Content View Mode so that if the count of the results was 1 or less, then it used a different view mode for the content than the default. Then I could use twig to theme to my heart's content.

However, I can't quite seem to make this work. I can correctly pull the number of rows from hook_views_pre_render, and while I can access the view mode (There are several permutations, but I suspect it's $view->rowPlugin->options['view_mode'] is correct), changing it during pre-render doesn't seem to have any effect).

Is there another hook I should be looking at? None of the other hook_views_pre_* seem to work. Is there another way of accomplishing what I'm trying to do? (Display a different view if the result count is a certain number).

How can I programatically alter the view mode of a view? (And can I do it after I have result rows?)

Thanks in advance!