Improve CSV export/import similarity

Created on 6 August 2017, over 7 years ago
Updated 18 April 2023, over 1 year ago

There are some major differences between the format of CSVs exported from farmOS and the format of CSVs that can be imported into farmOS. This makes it difficult to use the CSV export/import to move data from one farmOS to another.

Some of these differences are documented here: http://farmOS.org/guide/export

Due to these differences, it is not recommended to use CSVs for moving data from one farmOS to another.

We may be able to improve upon this...

One thing we can do is override the fields provided by the CSV exports, and differentiate them from the output that is displayed in the View's Page display. Some examples:

* Do not truncate description fields
* Output numerical asset IDs (instead of names)
* ...

✨ Feature request
Status

Closed: won't fix

Version

1.0

Component

Import/export

Created by

πŸ‡ΊπŸ‡ΈUnited States m.stenta

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡ΊπŸ‡ΈUnited States m.stenta

    I'm actually going to set this back to 7.x-1.x and close it as "won't fix".

    We aren't maintaining 7.x-1.x anymore, and we don't have CSV importers in 2.x (see #3151244: [META] farmOS 2.x Importers β†’ ). If we have the same issue after we add CSV importers, we can open a new issue.

  • Status changed to Postponed 11 months ago
  • πŸ‡ΊπŸ‡ΈUnited States m.stenta

    Update: We now have CSV importers in farmOS v2+ ( #3382539: farmOS v2 CSV import module β†’ ).

    I also just opened a pull request that will make the asset and log CSV exporters more closely aligned with the CSV importers: https://github.com/farmOS/farmOS/pull/783

    Regarding CSV export/import compatibility (issue (5) above), this is step in the right direction. It changes the CSV columns headers to be field IDs instead of field labels, which match what the CSV importers expect, and it includes more potential data. It may be worth reopening https://www.drupal.org/project/farm/issues/2900239 β†’ and documenting what all the differences are, so we can consider whether or not we want to continue the effort of bringing them together.

    There are still differences that prevent exports from being directly used in imports. Once that PR is merged, we can pick up this issue where we left off and document what all the differences are, and explore next steps to bring the two closer together.

    Reopening this issue, updating the summary, and marking it as "postponed" pending that PR...

  • πŸ‡ΊπŸ‡ΈUnited States m.stenta
  • Status changed to Active 9 months ago
  • πŸ‡ΊπŸ‡ΈUnited States m.stenta
  • πŸ‡ΊπŸ‡ΈUnited States m.stenta
  • πŸ‡ΊπŸ‡ΈUnited States m.stenta

    farmOS 3.1.0 and 3.3.0 changed the way that we generate CSV exports for assets, logs, and quantities (notably data stream exports have not changed, but are not relevant to this issue).

    See pull requests:

    To summarize, we used to provide CSV exports as View displays with a limited set of columns (the same columns as the page displays). Now, we provide an "Export CSV" bulk action which uses the CSV Serialization β†’ module to create CSV rows for selected entities.

    I referenced this issue in the first pull request:

    The CSV exports are not compatible with the CSV importers, and we basically decided they never will be (see https://www.drupal.org/project/farm/issues/2900239 β†’ ). This will still be the case after this PR, but it brings us a step closer to supporting this.

    Regarding CSV export/import compatibility (issue (5) above), this is step in the right direction. It changes the CSV columns headers to be field IDs instead of field labels, which match what the CSV importers expect, and it includes more potential data. It may be worth reopening https://www.drupal.org/project/farm/issues/2900239 β†’ and documenting what all the differences are, so we can consider whether or not we want to continue the effort of bringing them together.

    There are a couple of notable changes between the CSV files that were produced by the old code and this new code:

    1. CSV headers are field IDs instead of field labels.
    2. More columns are included.
    3. Quantity columns are NOT included (see "Next steps" below).
    4. Timestamp fields are returned in RFC3339 format.
    5. Multi-value cells use a pipe (|) separator instead of a comma (,).

    There may be others... those were the main ones I noticed/worked on personally. We get most of this output "for free" from the Drupal content entity normalizer, as opposed to specifying each one explicitly in Views, so we may find that there are bugs/issues that went unnoticed during my testing which we can address in follow-ups.

Production build 0.71.5 2024