Support views data export using drush

The above solution hijacks the views arguments for some internal flags, so it won't work for views that have contextual filters.
I re-wired the patch to move those flags into a different variable, and extended the drush command with a new parameter for specifying arguments.

So now it can be executed as follows:
drush vde [view_name] [display_name] [arg1/arg2/etc] [path/to/file.xml]

The last submitted patch, 76: vde-drush-with-output-location-2887450-v1.2-76.patch, failed testing. View results →
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

fixing errors and coding standards

Hello, I have tried this patch on my website and I am encountering mysql error. I am using module to export products from my drupal eshop and during the testing I have found out that using drush for small exports works fine. but with the larger exports I am always getting the error:
General error: 2006 MySQL server has gone away
I think it might be related to my hosting environment where is wait_timeout variable set only to 300 seconds. The thing is that this error is not showing up when running export through browser. In the browser I am redirected to batch page and export finish without problem. Throught browser I am able to export any number of products. Using the cli I get error with about 3000 products (I need to export about 50000 products).

Is there a way to resolve this error with cli command?

The solution I have found was to set this in settings.php file.

if (PHP_SAPI === 'cli') {
  ini_set('memory_limit', '200M');
}

For some reason it seems that drush is ignoring settings of php memory limit (some discussion here: https://github.com/drush-ops/drush/issues/3294) and is trying to run whole export in one process. After settings this limit the process consumes less then 200Mb of RAM and is working fine even with large exports. During export these messages are logged:

>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread

The last working patch for me is the one in comment #75 ✨ Support views data export using drush Needs review . None of the patches after that seem to save the export at the custom path defined.

I am using version 1.3 of the module which seems to be up to date with the dev version. The command I'm running is drush vde test_export data_export_1 "" private://test/test_export.csv. I have tried the public file system too, but still not saving at the path I set. In the end I am deleting the old file as part of the cron job, before running the drush command using the patch in #75 which works.

Patch re-roll for D10 (v1.3)

@PierreEmmanuel

The patch at #85 is missing the ViewsDataExportCommands.php class, is a mistake?

Meanwhile, I've updated the patch from #81, as on D10 emptying the caches gives the error:

PHP Fatal error:  Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20

Fatal error: Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20

Patch at #86 does not apply correctly for 8.x-1.3.
Re-rolled for compatibility.

Yes, I forgot to mention that #86 is against the dev version

The #86 as well the #87 missed deprecated a file_save_data - or better, the Rector missed it.

Attached a fixed version of #87

Hello,

First thanks for this patch, this features is really much appreciated.

I could be wrong, but I think the parameters order is inversed between outputfile and arguments in the viewsDataExport function. At least I add to inverse them for it to work proerly. Attached an updated patch with the right order.

@Giuseppe87 for #89:

Drupal 10.1.6, VDE 8.x-1.4

Patch applies however, in my testing it would appear that the copy of the file never happens to the final specified location.

It seems the $options array has a key in [1], however it is blank.
$args, however, does seem to hold the path in separate array indexes.

I've attached an updated full patch here.

~line 920 of src/Plugin/view/display/DataExport.php, I've altered my version to look like

else {
      // Due to https://github.com/drush-ops/drush/issues/5009
      // we need to set the $context['message'] instead of $context['results'].
      if (in_array('vde_drush', $options)) {
        $file_location = implode('/' , $args);
        $vde_file = \Drupal::service('file_system')->realpath($context['sandbox']['vde_file']);
        // If set, copy the file to the specified directory.
        if (array_key_exists(1, $options) && isset($file_location)) {
          if (!copy($vde_file, $file_location)) {
            $error_message = t('Could not write to final file location (@file). Check permissions.', ['@file' => $file_location]);
            \Drupal::logger('views_data_export')->error($error_message);
          }
          else {
            $context['sandbox']['vde_file'] = $file_location;
          }
        }
        $message = dt('Data export saved to !download_url', ['!download_url' => $context['sandbox']['vde_file']]);
        $context['message'] = $message;
      }

      // We're finished processing, set progress bar to 100%.
      $context['finished'] = 1;
    }

I've opened a MR based on patch from #90 and adjust this to recent supported Drush version 11/12 to make testing and reviewing easier for all of us.

Patch from #91 has strange format so it was not possible to apply for me.

I think we should also discuss with maintainers of https://www.drupal.org/project/vde_drush → module to merge Drush commands back into views_data_export code base.

Thanks for your work on this.

The patch from #90 applied smoothly using the patch command (not git). Tested on Drupal 10.2.26 + VDE 8.x-1.4.

Hoping this merge request gets approved quickly.

The MR needs the latest 8.x-1.x merged in as there is a conflict.

merged 8.x-1.x into this branch, though I will admit I only addressed the merge conflicts and haven't checked the rest of the patch in detail

Updated the merge request:

1. Modernized the drush command with attributes
2. Removed the drush.services.yml as this is longer used in Drush 12+
3. Fixed compatibility with Drupal 11 so PHPUnit runs correctly
4. General code cleanup

Merged 8.x-1.x into this branch, and fixed the merge conflicts.

I re-added the Drupal 11 compatibility fixes that got lost in the merge.

Hi, it seems that since the last update the patch (#90) and MR no longer work. Could it be updated?

I tried to fix it but there are many conflicts I have no context of the feature.

Thanks in advance.

Kind regards.

Hi, it seems that since the last update the patch (#90) and MR no longer work. Could it be updated?

I tried to fix it but there are many conflicts I have no context of the feature.

Thanks in advance.

Kind regards.

Or can the patch be merged? I am already successfully testing/running the patch on version 1.4 of Views Data Export on a production site.

I’ve merged the 8.x-1.x branch back into our MR.
I also fixed a PHPCS issue, but there are still other reported issues in the ViewsDataExportCommands.php file:
Namespaced classes/interfaces/traits should be referenced with use statements.

I left this issue unresolved, as it seems we don’t yet have a definitive decision on whether this is a concern. For reference, see 🐛 False Postive: Aliased PHP 8.0 Attributes on Class properties leading to "Namespaced classes/interfaces/traits should be referenced with use statements" Needs work .

Has anyone here tested this with D11 ?

steven jones → made their first commit to this issue’s fork.

The drush argument output_file doesn't work since the drush $options are being overridden by some views options.

Patch in attachment.
Diff agains merge request 34 in attachment.

Here is a patch for the current 8.x-1.5 release. Hiding the file.

I'm going to put this back into Needs work, and I'm going to assign it to myself, I'll work on it and get it to a place where I'm confortable committing it, ideally to 1.x, but maybe it'll go in a 2.x version.

I'm not going to commit this as-is because I don't think that the data export class should be making calls like drush_backend_batch_process if it can help it. Like, it seems weird that we're setting this 'your running in Drush' option and then changing quite a bit of the exection if that's the case. I'd rather find a way to isolate all of that to the Drush command if possible, providing enough output/hooks/events where needed for the Drush command to do what it needs to do.

I don't anticipate the actual invocation of the Drush command or the output changing between now and the final code that gets committed, so everyone here who's using the patch should be able to keep on using it and then do a trivial removal of the patch when it lands in a stable release of VDE.

Thanks for the patience everyone, should only be a little longer.

Thank you for your work.

But I have error on using patch #34 on line

// @phpstan-ignore-next-line
return \Drupal::service('file.repository')->writeData('', $destination, FileExists::Replace);

Error:
> > [26-May-2025 13:06:31 UTC] Error: Class "Drupal\Core\File\FileExists" not found in /var/www/html/modules/contrib/views_data_export/src/Plugin/views/display/DataExport.php on line 1078 #0 /var/www/html/modules/contrib/views_data_export/src/Plugin/views/display/DataExport.php(800): Drupal\views_data_export\Plugin\views\display\DataExport::getTempFile(Object(Drupal\views\ViewExecutable), 'content_pages_f...', 'data_export')
> > #1 /var/www/html/vendor/drush/drush/includes/batch.inc(257): Drupal\views_data_export\Plugin\views\display\DataExport::processBatch('content_pages_f...', 'data_export', Array, Array, 112100, Array, '', Array, Array)
> > #2 /var/www/html/vendor/drush/drush/includes/batch.inc(204): _drush_batch_worker()
> > #3 /var/www/html/vendor/drush/drush/includes/batch.inc(75): _drush_batch_command('329028')
> > #4 /var/www/html/vendor/drush/drush/src/Commands/core/BatchCommands.php(25): drush_batch_command('329028')
> > #5 [internal function]: Drush\Commands\core\BatchCommands->process('329028', Array)
> > #6 /var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php(276): call_user_func_array(Array, Array)
> > #7 /var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
> > #8 /var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php(175): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
> > #9 /var/www/html/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(387): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
> > #10 /var/www/html/vendor/symfony/console/Command/Command.php(326): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #11 /var/www/html/vendor/symfony/console/Application.php(1096): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #12 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #13 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #14 /var/www/html/vendor/drush/drush/src/Runtime/Runtime.php(110): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #15 /var/www/html/vendor/drush/drush/src/Runtime/Runtime.php(40): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #16 /var/www/html/vendor/drush/drush/drush.php(139): Drush\Runtime\Runtime->run(Array)
> > #17 /var/www/html/vendor/drush/drush/drush(4): require('/var/www/html/v...')
> > #18 /var/www/html/vendor/bin/drush(119): include('/var/www/html/v...')
> > #19 {main}

I have Drupal 10.2.8.

And another issue.

I execute command drush views_data_export:views-data-export content_pages_for_google_ads data_export and file
private://views_data_export\content_pages_for_google_ads_data_export\1-1748266716\filename.xml was created and never moved to the right place.

It's because in DataExport::processBatch() in line $options = $view->getStyle()->options; variable $options was overwritten.

One more thing.
On line $uri = $file->getFileUri();:

/var/www/html $ drush @rn views_data_export:views-data-export content_pages_for_google_ads data_export 345.xml
[notice] Starting data export..
> [warning] Undefined variable $file DataExport.php:938
> [26-May-2025 14:22:22 UTC] Error: Call to a member function getFileUri() on null in /var/www/html/modules/contrib/views_data_export/src/Plugin/views/display/DataExport.php on line 938

@sstapleton please see my changes in patch.

Small fix.

#34 (i.e. https://www.drupal.org/files/issues/2020-08-05/2887450-34_0.patch → ) is a very old patch... Are your patches based on the patch or the merge request?

@eelkeblok Sorry, I mean MR 34.

The following notices appears during the exporting to file:

/var/www/html/rn $ drush views_data_export:views-data-export view_id display_id file.xml
[notice] Starting data export ..
> [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
> [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
> [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
> [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
> [notice] Data export saved to private: //content_export/content-pages-for-google-ads/content-pages-for-google-ads.xml
[success] Data export finished.

Indeed memory consumption during batch calls grows. But this does not happen if you follow the link in the view and generate the file there. There are batch calls there too, but memory consumption does not grow at all.

steven jones → changed the visibility of the branch 2887450-re-apply-patch-to-add-drush-command to hidden.

Okay, I've done a bit of work on this one, removed the magic array of options and made it a proper classed object.

The major change that I've done here is to remove the account switching to user 1, which is kinda wrong for lots of reasons.

We could add an optional option to the Drush command to do this switching if that's what we want to do I think. Maybe one for a follow up?

Attached is a patch of the current MR!34 diff'd with 8.x-1.6 if people want to apply to it to that and see if it's still working for their use-case.

Correct patch name.

Hi Steven.

With views_data_export 1.6 and this patch on #128, I'm getting:

>  [warning] Undefined variable $file DataExport.php:927
>  [error]  Error: Call to a member function getFileUri() on null in Drupal\views_data_export\Plugin\views\display\DataExport::processBatch() (line 927 of /var/www/html/docroot/modules/contrib/views_data_export/src/Plugin/views/display/DataExport.php)

Line 797 in DataExport.php is probably not being run?

The command I'm using:

drush --root=/var/www/html/scripts/../docroot views_data_export:views-data-export doctor_data_export data_export_1 "/var/www/html/scripts/../docroot/sites/default/files/doc_exports/1749834394/doctor_export.csv"

I've done some further fixes in the MR, and attached a patch off of 1.6 for those who want to test like that.

FYI the code as it stands at the moment wants the output file to be a stream wrapper URI, which is different to the patches and code that has gone before, where it seemed like half the code wanted a stream wrapper and the other half wanted a plain file.

Given that the Drush code now does a bit of a weird thing where it optionally moves the file after generation to a user-specified location, I wonder if anyone could expand on their use-case for specifying the output file?
Are you using it to save it to locations outside of the Drupal private files directory?
Would that be a requirement?

I wonder about leaving the generation of the file alone, and then at the end of the Drush command, it can simply copy the file to the new location. That then seems more in line with my expectations of doing something random with the file at the end of the process tbh.

But, yeah, I'd like to hear the use-cases for this option.

Our use case is pretty simple: the file needs to be available at a specified location, under a specific, descriptive name. We are actually generating the file in the private file location, although I could imagine the need to put it somewhere else. I'm not sure what you mean by "I wonder about leaving the generation of the file alone, and then at the end of the Drush command, it can simply copy the file to the new location." Do you mean to keep the different name out of the scope of this Drush command and leave it to the caller to move the file afterwards? In our case, that would make things more complicated, as we currently have a single line in our crontab to generate the file in a specified location using this command. I suppose it is possible to do that all in a single line in a crontab, but I'd also say that allowing the user to specify a file name for the export file is a very reasonable feature to have.

@eelkeblok: thanks for letting me know your use-case.

I wonder about leaving the generation of the file alone, and then at the end of the Drush command, it can simply copy the file to the new location.

To hopefully clarify what I mean: in the MR there's some code that watches for the end of the batch processing, and at the end changes the location of the file that VDE generates during normal processing, and updates Drupal. So Drupal is fully aware of the file, it's in the manage file page, Drupal will delete the temporary file on cron etc.
This seems like a bit of a kludge to me.

I am wondering about leaving the location of the temporary file alone, so that we don't need to add more code to the regular display handler, but at the end of the Drush command code we could take a copy of the file from the Drupal managed filesystem and copy it somewhere else that Drush has access to, which doesn't need to then use stream wrappers etc. it would basically be a plain file-copy, with no associated clean-up semantics etc.

In our particular case, the output file does not need to be managed by Drupal because it's overwritten every night when a new version is generated so a plain PHP file-copy works for us. We generate the output file every night so that another process from another vendor can log into the server, get the file from a predefined location and process it.

Just sharing my setup in case it’s helpful to others.

In my case, I run the Drush export every night via crontab. The export generates a fresh XML file, which is saved under the same filename to a specific location on the server, overwriting the one from the night before. It works reliably, and I’m still using the patched VDE 1.5 version.

Here’s the cron job I use:

15 3 * * * cd /data/sites/web/www && /usr/local/bin/drush vde publication_export data_1 /data/sites/web/www/client/file.xml --force >> /data/sites/web/www/client/file_export.log 2>&1

@steven jones Thanks for that clarification. Our use case does not rely on the file being managed, so your proposal sounds fine.

Okay, so in summary I believe that the things left to do are:

• Re-work the Drush file saving stuff to make an unmanaged copy at the end of the export process.
• Add back in the weird 'run this as another user' stuff that was unconditionally running exports as UID1

Here's an updated patch, if people want to give the latest code a spin.

I've done those two items now, so we copy the file after the usual export process runs, and we allow for changing the account that's doing the export via Drush.

Okay, just pushed some semi-major changes to the Drush command, mostly internal, one big external change:

It'll now output the file on stdout if you don't specify an output file. That's the major difference.

I'm also tempted to change that as an option, rather than an argument, because that makes much more sense to me. That will introduce a B/C break with the previous patches, but not a crazy big one, so I'm might let myself get away with it.

Much happier with the code now. The changes to the Display class are largely cosmetic and it doesn't have any Drush specific code in it any more, which is also ace.

I think this is good to merge now.

Going to pop this into a 1.x branch and see what happens when I release, since it bumps the PHP version required to 8.1, which we should probably do in a major version, but hey ho, we're not Semver!