Contrib.social

drush wfx exporters yaml and json produce unusable, binary output

Open on Drupal.org →
Created on 19 September 2023, over 1 year ago
Updated 8 April 2024, about 1 year ago

Problem/Motivation

YAML and JSON output with drush wfx --exporter=yaml or --exporter=json should do what they advertise.
Instead, some encoded binary blob gets exported.
Coreutils' file says
gzip compressed data, extra field, has comment, last modified: Thu Feb 7 16:27:08 2019, max compression, from Unix, original size modulo 2^32 173825160
but gzip does not like it:
gzip: stdin has flags 0x5c -- not supported
The webGUI-based exporter works correctly and gives me a .tar.gz or a .zip file, so basically, a compressed file can be expected (but should be noted in the usage screen that this is no direct .json or .yml).

I hexdump'd and diff'd the webGUI's output and the drush output, but they differ largely.

I also found https://github.com/drush-ops/drush/issues/4026 and thought that there might probably be some other (plain text) output in the .gz file that somehow sneaked in, but that's not the case. Still, it sounds like a possible cause that there might be some unrelated string that gets compressed with the actual output.

If I add $this->configuration['archive_type'] = 'zip'; to src/Plugin/WebformExporter/JsonWebformExporter.php right before the call to $this->addToArchive($json, $file_name);, I get an error when I run 

$ vendor/bin/drush wfx myform --exporter=json
>  [notice] Exported 1 of 1 submissions…
 [warning] file_get_contents(/tmp/myform.tar.gz): Failed to open stream: No such file or directory WebformSubmissionCommands.php:183

However, if I open /tmp/myform.zip then, I can see the correct json.

If I try to add --archive-type=zip directly, the output is broken as well, which is pretty strange.

Something in the compression step seems to go wrong.

Steps to reproduce

Use drush wfx --exporter=json or drush wfx --exporter=yaml to get an invalid tar.gz file

Proposed resolution

Fix compression.

Probably just really export json or yaml when configured to do so and don't put it in an archive. Make archiving optional, if it is helpful for attachments etc.
But at least state you get a compressed tarball (see User interface changes below.

User interface changes

Add a note to the usage that "tar" means "tar.gz".
Or better yet, call it "tar.gz".

🐛 Bug report
Status

Active

Version

6.2

Component

Code

Created by

🇦🇹Austria tgoeg

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @tgoeg
  • Status changed to Closed: works as designed about 1 year ago
  • Comment about 1 year ago →
    🇺🇸United States jrockowitz Brooklyn, NY

    You can choose the archive type in the UI and drush command.
    →

  • Status changed to Active about 1 year ago
  • Comment about 1 year ago →
    🇦🇹Austria tgoeg

    Yes, as stated in my original report, the GUI setting you show works indeed as it should.

    However, drush does not produce anything useful when using --exporter=json or --exporter=yaml !

    Tested with current v6.2.2

    Do you get any meaningful output when you issue the following?

    $ vendor/bin/drush wfx myform --exporter=json > /tmp/exporttest
>  [notice] Exported 441 of 441 submissions…
$ file /tmp/exporttest
/tmp/exporttest: gzip compressed data, extra field, has comment, last modified: Thu Feb  7 16:27:08 2019, max compression, from Unix, original size modulo 2^32 173806772
$ gzip -dc /tmp/exporttest | less
gzip: /tmp/exporttest has flags 0x5c -- not supported

    If I substitute the 0x5c in the file with 0x00 I can tar xzf /tmp/exporttest and get a file submission-3.json which is half broken, but still it shows it is JSON in (broken) .tar.gz

    1) The output is unusable
    2) The output is compressed when the option/usage info does not say so. Either do output JSON/YAML as one would expect or state you get a .tar.gz (which should not be corrupt :-) )

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024