Excluding all that are not in sitemap

When hackers/bots visit our pages they attempt query strings and those are being cached.
This generates TONS of files. (we use file caching)

Is there a way to limit the cache to the files that are in our sitemap? And exclude the others? So the system does not generate all those.