Is this possible to use only Service workers?

Yes, using the SW for performance gains without deploying the manifest file will avoid any chance of a PWA prompt.

I've never tried it, but it might be possible to use a theme hook to unset the manifest markup from your HTML output: It is set inside pwa_page_attachments: https://git.drupalcode.org/project/pwa/-/blob/8.x-1.x/pwa.module#L37

Thanks