Contrib.social

Support Batch Processing

Open on Drupal.org →
Created on 3 April 2025, 16 days ago

Problem/Motivation

The FileMakerApiFetcher class lacks batching support, potentially causing timeouts when fetching large datasets.

Steps to reproduce

  1. Create a Feeds importer using FileMakerApiFetcher.
  2. Configure it to fetch a large dataset.
  3. Run the import process.
  4. Observe timeout or failure for large datasets.

Proposed resolution

Implement batching using '_limit' and '_offset' parameters as per the FileMaker Data API Guide.

Remaining tasks

  1. Implement batching logic in FileMakerApiFetcher.
  2. Add batch size configuration option.
  3. Update fetch method for batching.
  4. Implement progress reporting.
  5. Write unit tests.
  6. Update documentation.
  7. Perform thorough testing.

User interface changes

  • Add batch size field in fetcher configuration form.
  • Update progress indicator for batch processing.

API changes

  • Modify FileMakerApiFetcher::fetch() to support batching.
  • Add new protected methods for batch processing.
  • Update relevant hooks/events.

Data model changes

No direct changes. Temporary data storage during import may be affected.

✨ Feature request
Status

Active

Version

2.0

Component

Code

Created by

🇧🇪Belgium baikho Antwerp, BE

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024