Does not work with image_svg module

A check wouldn't hurt, but you shouldn't use this formatter for svg's.

Mixing svg and pixel based formats in the same field is not a good idea.
Does core's responsive image support this?

Not sure if core supports this. For animated GIFs there is a separate module to omit image processing on GIFs: https://www.drupal.org/project/animated_gif → - maybe to would be possible to "borrow" some code from there to make it work for SVGs too in this module?