Uploading larger file than the memory limit fails

Open on Drupal.org →

Created on 13 June 2023, about 1 year ago

Updated 26 November 2023, 10 months ago

Problem/Motivation

Uploading large files that are potentially bigger than the allowed memory limit can ran into out of memory problems, causing the file upload to fail.

Steps to reproduce

Set PHP upload_max_filesize and post_max_size to 550M.
Set PHP memory_limit to 256M.
Ensure that your webserver can accept a file with 500 MB. (In nginx it can be set as client_max_body_size 550M; in either http, server or location block.)
Install Drupal with standard profile.
Add a new file field to a node with the maximum upload size of 550 MB.
Generate a test file (dd if=/dev/urandom of=file.txt bs=500M count=1, also the size of the file should be somewhat less than the allowed in relation to Drupal, php and webserver).
Try to upload the generated file to the file field.
In my test setup there was no error message nor log entries, so the upload just silently fails, saving the node the file won't be saved.
After upload the ajax response should contain the error message, something like: Allowed memory size of 268435456 bytes exhausted (tried to allocate 524288032 bytes).

Proposed resolution

Replace move_uploaded_file() with stream_copy_to_stream() in \Drupal\Core\File\FileSystem::moveUploadedFile().
If you check some public FOSS projects, like NextCloud's source code, you can see this is a widely used solution for moving files in a memory friendly way: https://github.com/search?q=repo%3Anextcloud%2Fserver+stream_copy_to_str...

Alternative considerations

rename() was also considered as an alternative solution but after some research, it turned out that Symfony used to use rename() but they replaced it because rename() worked inconsistently when the source and target filesystem was different (like when the target was NFS.

https://github.com/symfony/symfony/commit/447ff915df4dc896bb5e2f7dc069c9... ---> related PR: https://github.com/symfony/symfony/pull/6211

Related issues:
https://github.com/symfony/symfony/pull/5878
https://github.com/symfony/symfony/issues/6185

Remaining tasks

See #12 for question about adding tests.

User interface changes

API changes

Data model changes

Release notes snippet

🐛 Bug report

Status

Needs work

Version

11.0 🔥

Component

File system →

Last updated 1 day ago

Maintained by
🇦🇺Australia @kim.pepper

Created by

🇭🇺Hungary yce

Live updates comments and jobs are added and updated live.

Sign in to follow issues

Merge Requests

!4169Uploading larger file than the memory limit fails
Open
🇭🇺Hungary yce
updated 8 months ago

Comments & Activities

Issue created by @yce
Assigned to Anchal_gupta
Comment about 1 year ago →
🇮🇳India Anchal_gupta
Comment about 1 year ago →
🇮🇳India Anchal_gupta
I have uploaded the patch. Please review
Issue was unassigned.
Status changed to Needs review about 1 year ago9:49am 13 June 2023
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.1 & MySQL 5.7
last update about 1 year ago
28,330 pass, 92 fail
Comment about 1 year ago →
🇮🇳India Anchal_gupta
Merge request !4169Issue #3366403: Uploading larger file than the memory limit fails → (Open) created by yce
Open on Drupal.org →
Environment: PHP 8.1 & MySQL 5.7
last update about 1 year ago
Not currently mergeable.
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.1 & MySQL 5.7
last update about 1 year ago
28,485 pass, 6 fail
Comment about 1 year ago →
🇭🇺Hungary yce
Hiding incorrect patch as the stream_copy_to_stream() requires resource for the first two arguments, not filename strings.
https://www.php.net/manual/en/function.stream-copy-to-stream.php
Comment about 1 year ago →
System Message
The last submitted patch, 3: 3366403.patch, failed testing. View results →
Status changed to Needs work about 1 year ago11:13am 13 June 2023
Comment about 1 year ago →
🇺🇸United States cilefen
Comment about 1 year ago →
🇭🇺Hungary mxr576 Hungary
Hiding incorrect patch as the stream_copy_to_stream() requires resources for the first two arguments, not filename strings.

Indeed, I am sorry to say, but I am uncertain if @Anchal_gupta your immediate (within 3 minutes :O ) contribution is worth a credit.
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.1 & MySQL 5.7
last update about 1 year ago
28,485 pass, 6 fail
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.1 & MySQL 5.7
38:53
35:44
Running
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.1 & MySQL 5.7
last update about 1 year ago
28,523 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.1 & MySQL 5.7
last update about 1 year ago
28,526 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.1 & MySQL 5.7
last update about 1 year ago
28,526 pass
Comment about 1 year ago →
🇭🇺Hungary mxr576 Hungary
@yce please also change the MR target to 11.x branch and rebase if necessary.
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,448 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,448 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,448 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,448 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,448 pass
Comment about 1 year ago →
🇭🇺Hungary mxr576 Hungary
Status changed to RTBC about 1 year ago3:40pm 14 June 2023
Comment about 1 year ago →
🇭🇺Hungary mxr576 Hungary
LGTM, I am uncertain what kind of test could be added to cover this problem. Maybe others have some smart idea.
Comment about 1 year ago →
🇭🇺Hungary yce
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,486 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,499 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,499 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,531 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,553 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,554 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,563 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,571 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,571 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,801 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,802 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,802 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,806 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,811 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,815 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,815 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,822 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,827 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,874 pass, 2 fail
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,879 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,884 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,887 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,911 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,934 pass, 2 fail
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,952 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,953 pass
Comment about 1 year ago →
🇳🇿New Zealand quietone New Zealand
I am doing triage on the core RTBC queue → .

The issue is missing remaining tasks (which is handy for reviewers) but does have a proposed resolution. And reading the patch the resolution seems to fit the code changes. There are reviews of the code as well.

In #12 there is a question about tests. I can't answer that but I have added the 'remaining tasks' and put that in there.

Added credit for those I was sure about.
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,953 pass
Comment about 1 year ago →
🇧🇷Brazil aluzzardi Pelotas, RS
I can confirm that this error also happens on Drupal 9.5.10
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,958 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,958 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,959 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
29,961 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
30,047 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
30,056 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
30,056 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
30,058 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
57:01
52:42
Running
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
30,060 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
30,100 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
30,135 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
30,135 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update about 1 year ago
30,136 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,146 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,146 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,148 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,157 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,161 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,164 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,168 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,168 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,205 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,363 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 12 months ago
30,361 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,360 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,362 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,379 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,377 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,382 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,392 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
26:54
21:18
Running
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,397 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,413 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,417 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,425 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,426 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,436 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 11 months ago
30,438 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,464 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,481 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,482 pass, 1 fail
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,486 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,486 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,510 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
Build Successful
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,519 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,530 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,560 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,602 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,604 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,605 pass
Open in Jenkins → Open on Drupal.org →
Environment: PHP 8.2 & MySQL 8
last update 10 months ago
30,668 pass
Status changed to Needs work 10 months ago12:50pm 26 November 2023
Comment 10 months ago →
🇬🇧United Kingdom alexpott 🇪🇺🌍
I've added a question and review points to the MR.

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024