Files in optimized directory not used

I'm trying to understand how advagg works. I can see that it creates a lot of files in files/js/optimized but these files are not referenced in the HTML unless I also disable core aggregation.

If I disable core aggregation and enable bundler the files are still not aggregated (bundled) and are still served from the optimized directory.

What is the correct way to get optimized files created by advagg and served in an aggregated way? What I want is the gzipped files files to be served and cache headers set correctly which currently is not happening using core files.