Memory limit issue with large JSON files.

Created on 26 August 2024, 5 months ago

Problem/Motivation

When attempting to process large JSON files we are running into memory limit due to JsonSource.php needing to load the entire file into memory for JSON decoding.

Steps to reproduce

- PHP memory limit 400MB
- Add pipeline and add a JSON file that is 80MB.
- Should fail during it's attempt to process.

Proposed resolution

- Update JsonSource.php to use JSON streaming using the package halaxa/json-machine.
- This ensures we don't run into memory limit issues.

Remaining tasks

- Review patch

🐛 Bug report
Status

Active

Version

1.0

Component

Code

Created by

🇦🇺Australia cman9090

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024