How to load multiple files conditionally?

Is it possible to describe a library (using hook_libraries_info()) that includes multiple files, but in such a way that you can load the individual files conditionally?

Take, for example, the Flot library: https://github.com/flot/flot
The main library file is jquery.flot.js however, depending on the functionality you want, you may also need to load the jquery.flot.time.js and/or jquery.flot.symbol.js files. You don't want the last two files loaded unless a certain condition is met (e.g. the user selects an option that requires one of these files).

The only ways I can see are either listing all files in the library (in which case it seems they'll all be loaded regardless), or creating separate libraries for each file (which seems unnecessary since most of the info for each library would be the same (name, download URL, version, etc.)).