Support views data export using drush

The above solution hijacks the views arguments for some internal flags, so it won't work for views that have contextual filters.
I re-wired the patch to move those flags into a different variable, and extended the drush command with a new parameter for specifying arguments.

So now it can be executed as follows:
drush vde [view_name] [display_name] [arg1/arg2/etc] [path/to/file.xml]

The last submitted patch, 76: vde-drush-with-output-location-2887450-v1.2-76.patch, failed testing. View results →
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

fixing errors and coding standards

Hello, I have tried this patch on my website and I am encountering mysql error. I am using module to export products from my drupal eshop and during the testing I have found out that using drush for small exports works fine. but with the larger exports I am always getting the error:
General error: 2006 MySQL server has gone away
I think it might be related to my hosting environment where is wait_timeout variable set only to 300 seconds. The thing is that this error is not showing up when running export through browser. In the browser I am redirected to batch page and export finish without problem. Throught browser I am able to export any number of products. Using the cli I get error with about 3000 products (I need to export about 50000 products).

Is there a way to resolve this error with cli command?

The solution I have found was to set this in settings.php file.

if (PHP_SAPI === 'cli') {
  ini_set('memory_limit', '200M');
}

For some reason it seems that drush is ignoring settings of php memory limit (some discussion here: https://github.com/drush-ops/drush/issues/3294) and is trying to run whole export in one process. After settings this limit the process consumes less then 200Mb of RAM and is working fine even with large exports. During export these messages are logged:

>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread

The last working patch for me is the one in comment #75 ✨ Support views data export using drush Needs review . None of the patches after that seem to save the export at the custom path defined.

I am using version 1.3 of the module which seems to be up to date with the dev version. The command I'm running is drush vde test_export data_export_1 "" private://test/test_export.csv. I have tried the public file system too, but still not saving at the path I set. In the end I am deleting the old file as part of the cron job, before running the drush command using the patch in #75 which works.

Patch re-roll for D10 (v1.3)

@PierreEmmanuel

The patch at #85 is missing the ViewsDataExportCommands.php class, is a mistake?

Meanwhile, I've updated the patch from #81, as on D10 emptying the caches gives the error:

PHP Fatal error:  Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20

Fatal error: Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20

Patch at #86 does not apply correctly for 8.x-1.3.
Re-rolled for compatibility.

Yes, I forgot to mention that #86 is against the dev version

The #86 as well the #87 missed deprecated a file_save_data - or better, the Rector missed it.

Attached a fixed version of #87

Hello,

First thanks for this patch, this features is really much appreciated.

I could be wrong, but I think the parameters order is inversed between outputfile and arguments in the viewsDataExport function. At least I add to inverse them for it to work proerly. Attached an updated patch with the right order.

@Giuseppe87 for #89:

Drupal 10.1.6, VDE 8.x-1.4

Patch applies however, in my testing it would appear that the copy of the file never happens to the final specified location.

It seems the $options array has a key in [1], however it is blank.
$args, however, does seem to hold the path in separate array indexes.

I've attached an updated full patch here.

~line 920 of src/Plugin/view/display/DataExport.php, I've altered my version to look like

else {
      // Due to https://github.com/drush-ops/drush/issues/5009
      // we need to set the $context['message'] instead of $context['results'].
      if (in_array('vde_drush', $options)) {
        $file_location = implode('/' , $args);
        $vde_file = \Drupal::service('file_system')->realpath($context['sandbox']['vde_file']);
        // If set, copy the file to the specified directory.
        if (array_key_exists(1, $options) && isset($file_location)) {
          if (!copy($vde_file, $file_location)) {
            $error_message = t('Could not write to final file location (@file). Check permissions.', ['@file' => $file_location]);
            \Drupal::logger('views_data_export')->error($error_message);
          }
          else {
            $context['sandbox']['vde_file'] = $file_location;
          }
        }
        $message = dt('Data export saved to !download_url', ['!download_url' => $context['sandbox']['vde_file']]);
        $context['message'] = $message;
      }

      // We're finished processing, set progress bar to 100%.
      $context['finished'] = 1;
    }

I've opened a MR based on patch from #90 and adjust this to recent supported Drush version 11/12 to make testing and reviewing easier for all of us.

Patch from #91 has strange format so it was not possible to apply for me.

I think we should also discuss with maintainers of https://www.drupal.org/project/vde_drush → module to merge Drush commands back into views_data_export code base.

Thanks for your work on this.

The patch from #90 applied smoothly using the patch command (not git). Tested on Drupal 10.2.26 + VDE 8.x-1.4.

Hoping this merge request gets approved quickly.

The MR needs the latest 8.x-1.x merged in as there is a conflict.

merged 8.x-1.x into this branch, though I will admit I only addressed the merge conflicts and haven't checked the rest of the patch in detail

Updated the merge request:

1. Modernized the drush command with attributes
2. Removed the drush.services.yml as this is longer used in Drush 12+
3. Fixed compatibility with Drupal 11 so PHPUnit runs correctly
4. General code cleanup

Merged 8.x-1.x into this branch, and fixed the merge conflicts.

I re-added the Drupal 11 compatibility fixes that got lost in the merge.