[meta] Rethink Data Export from the ground up

Views Data Export grew out of a need for a specific client and forked from Views Bonus Pack, and has steadily grown and been added to. The core of the code and module however has changed very little since the start, time for a re-think.

There are a number of dark corners of the module where we do some scary things. A number of these were in because when this was originally written for Drupal 6, they made sense, or seemed quite clever at the time. They are now just legacy cruft we should remove.

Way back when we did architect the module a little bit, but basically we were working entirely within a view, and the views systems. Views doesn't actually fit some of the requirements of Views Data Export, we've shoehorned them in, but that's probably not for the best really.

We should stop, take a look at our requirements, and come up with sensible solutions and progress that work forward.

Requirements

• Export data into various file formats: xml, csv, json, xslx.
• Fetch data from Views.
• Provide Drush integration to build the export.
• Be able to work with massive datasets and export them consistently.
• Be maintainable and have a sensible code structure and layout.
• Be testable, and tested
• Have a nice path to Drupal 8

Potential architectures

Exporters

The 7.x-4.x branch has lots of lovely code inspired by webform that can export to a proper xslx file, and punts the job of actually building an export file into unit-testable classes. I like this, I like this a lot. This is better than what we do in 7.x-3.x, which is using Drupal's theme system, which isn't in any way unit testable.
It also allows a small and defined integration point for people wanting to change or extend how data is exported. There are PHP extensions for writing xlsx files that can do crazy things, we should support using them by just extending a class and integrating a few methods.

I think that in principle exporters could be spun out into their own module, and modules like webform, could add them as a dependency, you know, if webform depending on anything was ever a thing.

Sources

I'd like to support grabbing data from views, that's the heart of this module, and covers many, many use cases, but we should have an abstract concept of a source of data, and the the View source can just be a specific implementation of that.

Once we have an abstract notion of something that gives us data, then we could write wrappers for other things, like standard SQL queries, or EntityFieldQueries or something, then if you're writing a module and want to provide a quick XLSX export, you don't have to set up a view and this module, you can just throw us something that wraps the source, and let Data Export handle the complexity of wrangling that into the appropriate format.

UIs

Once we have Exporters and Sources, then I think we just lack some UIs to glue it all together. One of these would probably attach itself to views, and allow configuration through the views UI, and one would be accessed through Drush.

Other things

We'll probably need other things that I've not thought of yet.

Feel free to comment/discuss and change the issue summary.