Batch export is not working correctly

Created on 4 December 2022, over 1 year ago
Updated 21 December 2023, 6 months ago

Thanks for great module!
I have some suggestions to imporove it.

Problem/Motivation

Batch export is not work correctly:
- it doesnt provide correct amount of results. For example, limit is 95, items per batch 10. It will export only 90.
- offset is not set correctly.
- not correct condition to check how many items in queue. It will not work if you have a couple of exports.
- queue api will not work correct if there are more than 1 batch export exists.

Proposed resolution

provided patch will fix:
handling correct amount of items in batch export.
Remove not necassaryconditions to check $queue->numberOfItems() and current batch total items.
Using view_id and display_id in queue worker. This will prevent building the same view twice. I've made a check and previous solution with settings offset to queue was not working for me.

Data model changes

I've change the data we added to queue item from view object to view_id and display_id.

πŸ› Bug report
Status

RTBC

Version

2.0

Component

Code

Created by

πŸ‡ΊπŸ‡¦Ukraine andriic

Live updates comments and jobs are added and updated live.
  • views

    Involves, uses, or integrates with views. In Drupal 8 core, use the β€œVDC” tag instead.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡ΊπŸ‡¦Ukraine andriic

    Updated patch. I found an issue where in exported XML file last close tag was missed.

  • Status changed to Needs work about 1 year ago
  • πŸ‡¨πŸ‡¦Canada smulvih2 Canada 🍁

    @andriic thank you very much for your work on this!

    Can we please split this into two tickets? Let's use this ticket to fix the offset issue, and open a new ticket to deal with the queue issue(s). I will need some additional info on how to reproduce the last two issues from the ticket description:

    - not correct condition to check how many items in queue. It will not work if you have a couple of exports.
    - queue api will not work correct if there are more than 1 batch export exists.

    I have a site using multiple batch exports and it seems to work correctly for me, so interested in how to reproduce. Thanks!

  • πŸ‡ΊπŸ‡¦Ukraine andriic

    Hi @smulvih2.

    Incorrect rows in results.
    Steps to reproduce bug with incorrect offset:
    1) Create new view with CSV data export.
    2) set path, filename. Export Method: Batch, size 100, limit 101.
    3) export with drush command
    drush vde_drush:views-data-export view_name display_id /filepath/test.csv --yes

    Result: in exported file you will have header more than 200 rows.

    As I previosuly mentioned, I've changed data in queue item from views object to view_id and display_id.
    This will allow use to execute query each time with correct offset and items per page.

    Issue with queue items:
    Queue can consists of more then one export when it's running in the same time.
    For example you have a cron which use drush to export and at the same time you doing it manually from drush command.
    That's why I've removed line
    - $total_queue_items = $queue->numberOfItems() * $items_per_batch - $items_per_batch;
    which makes no sense if there are more than one batch at the same time.

  • Status changed to Needs review 12 months ago
  • Status changed to RTBC 6 months ago
  • πŸ‡ͺπŸ‡ΈSpain pcambra Spain, πŸ‡ͺπŸ‡Ί
Production build 0.69.0 2024