Support views data export using drush

The above solution hijacks the views arguments for some internal flags, so it won't work for views that have contextual filters.
I re-wired the patch to move those flags into a different variable, and extended the drush command with a new parameter for specifying arguments.

So now it can be executed as follows:
drush vde [view_name] [display_name] [arg1/arg2/etc] [path/to/file.xml]

The last submitted patch, 76: vde-drush-with-output-location-2887450-v1.2-76.patch, failed testing. View results →
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

fixing errors and coding standards

Hello, I have tried this patch on my website and I am encountering mysql error. I am using module to export products from my drupal eshop and during the testing I have found out that using drush for small exports works fine. but with the larger exports I am always getting the error:
General error: 2006 MySQL server has gone away
I think it might be related to my hosting environment where is wait_timeout variable set only to 300 seconds. The thing is that this error is not showing up when running export through browser. In the browser I am redirected to batch page and export finish without problem. Throught browser I am able to export any number of products. Using the cli I get error with about 3000 products (I need to export about 50000 products).

Is there a way to resolve this error with cli command?

The solution I have found was to set this in settings.php file.

if (PHP_SAPI === 'cli') {
  ini_set('memory_limit', '200M');
}

For some reason it seems that drush is ignoring settings of php memory limit (some discussion here: https://github.com/drush-ops/drush/issues/3294) and is trying to run whole export in one process. After settings this limit the process consumes less then 200Mb of RAM and is working fine even with large exports. During export these messages are logged:

>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread

The last working patch for me is the one in comment #75 ✨ Support views data export using drush Needs review . None of the patches after that seem to save the export at the custom path defined.

I am using version 1.3 of the module which seems to be up to date with the dev version. The command I'm running is drush vde test_export data_export_1 "" private://test/test_export.csv. I have tried the public file system too, but still not saving at the path I set. In the end I am deleting the old file as part of the cron job, before running the drush command using the patch in #75 which works.

Patch re-roll for D10 (v1.3)

@PierreEmmanuel

The patch at #85 is missing the ViewsDataExportCommands.php class, is a mistake?

Meanwhile, I've updated the patch from #81, as on D10 emptying the caches gives the error:

PHP Fatal error:  Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20

Fatal error: Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20

Patch at #86 does not apply correctly for 8.x-1.3.
Re-rolled for compatibility.

Yes, I forgot to mention that #86 is against the dev version

The #86 as well the #87 missed deprecated a file_save_data - or better, the Rector missed it.

Attached a fixed version of #87

Hello,

First thanks for this patch, this features is really much appreciated.

I could be wrong, but I think the parameters order is inversed between outputfile and arguments in the viewsDataExport function. At least I add to inverse them for it to work proerly. Attached an updated patch with the right order.

@Giuseppe87 for #89:

Drupal 10.1.6, VDE 8.x-1.4

Patch applies however, in my testing it would appear that the copy of the file never happens to the final specified location.

It seems the $options array has a key in [1], however it is blank.
$args, however, does seem to hold the path in separate array indexes.

I've attached an updated full patch here.

~line 920 of src/Plugin/view/display/DataExport.php, I've altered my version to look like

else {
      // Due to https://github.com/drush-ops/drush/issues/5009
      // we need to set the $context['message'] instead of $context['results'].
      if (in_array('vde_drush', $options)) {
        $file_location = implode('/' , $args);
        $vde_file = \Drupal::service('file_system')->realpath($context['sandbox']['vde_file']);
        // If set, copy the file to the specified directory.
        if (array_key_exists(1, $options) && isset($file_location)) {
          if (!copy($vde_file, $file_location)) {
            $error_message = t('Could not write to final file location (@file). Check permissions.', ['@file' => $file_location]);
            \Drupal::logger('views_data_export')->error($error_message);
          }
          else {
            $context['sandbox']['vde_file'] = $file_location;
          }
        }
        $message = dt('Data export saved to !download_url', ['!download_url' => $context['sandbox']['vde_file']]);
        $context['message'] = $message;
      }

      // We're finished processing, set progress bar to 100%.
      $context['finished'] = 1;
    }

I've opened a MR based on patch from #90 and adjust this to recent supported Drush version 11/12 to make testing and reviewing easier for all of us.

Patch from #91 has strange format so it was not possible to apply for me.

I think we should also discuss with maintainers of https://www.drupal.org/project/vde_drush → module to merge Drush commands back into views_data_export code base.

Thanks for your work on this.

The patch from #90 applied smoothly using the patch command (not git). Tested on Drupal 10.2.26 + VDE 8.x-1.4.

Hoping this merge request gets approved quickly.

The MR needs the latest 8.x-1.x merged in as there is a conflict.

merged 8.x-1.x into this branch, though I will admit I only addressed the merge conflicts and haven't checked the rest of the patch in detail

Updated the merge request:

1. Modernized the drush command with attributes
2. Removed the drush.services.yml as this is longer used in Drush 12+
3. Fixed compatibility with Drupal 11 so PHPUnit runs correctly
4. General code cleanup

Merged 8.x-1.x into this branch, and fixed the merge conflicts.

I re-added the Drupal 11 compatibility fixes that got lost in the merge.

Hi, it seems that since the last update the patch (#90) and MR no longer work. Could it be updated?

I tried to fix it but there are many conflicts I have no context of the feature.

Thanks in advance.

Kind regards.

Hi, it seems that since the last update the patch (#90) and MR no longer work. Could it be updated?

I tried to fix it but there are many conflicts I have no context of the feature.

Thanks in advance.

Kind regards.

Or can the patch be merged? I am already successfully testing/running the patch on version 1.4 of Views Data Export on a production site.

I’ve merged the 8.x-1.x branch back into our MR.
I also fixed a PHPCS issue, but there are still other reported issues in the ViewsDataExportCommands.php file:
Namespaced classes/interfaces/traits should be referenced with use statements.

I left this issue unresolved, as it seems we don’t yet have a definitive decision on whether this is a concern. For reference, see 🐛 False Postive: Aliased PHP 8.0 Attributes on Class properties leading to "Namespaced classes/interfaces/traits should be referenced with use statements" Needs work .

Has anyone here tested this with D11 ?

steven jones → made their first commit to this issue’s fork.

The drush argument output_file doesn't work since the drush $options are being overridden by some views options.

Patch in attachment.
Diff agains merge request 34 in attachment.

Here is a patch for the current 8.x-1.5 release. Hiding the file.

I'm going to put this back into Needs work, and I'm going to assign it to myself, I'll work on it and get it to a place where I'm confortable committing it, ideally to 1.x, but maybe it'll go in a 2.x version.

I'm not going to commit this as-is because I don't think that the data export class should be making calls like drush_backend_batch_process if it can help it. Like, it seems weird that we're setting this 'your running in Drush' option and then changing quite a bit of the exection if that's the case. I'd rather find a way to isolate all of that to the Drush command if possible, providing enough output/hooks/events where needed for the Drush command to do what it needs to do.

I don't anticipate the actual invocation of the Drush command or the output changing between now and the final code that gets committed, so everyone here who's using the patch should be able to keep on using it and then do a trivial removal of the patch when it lands in a stable release of VDE.

Thanks for the patience everyone, should only be a little longer.