Follow up from issue 3283364 as well as comment #4 from this issue's reporter.
The merged commit (94a2ab3d) contains a fix for ExtraFieldBlockPlugin, but it appears ExtraFieldViewsPlugin also has an issue.
The problem is that a ViewExecutable will build a render array regardless of whether it has anything to display, because the view is not executed until render time, and hence it is not sufficient to check the render array.
You have to actually execute the view with arguments, then check $view->getDisplay()->outputIsEmpty()