Random failure: "PHP temp directory (/tmp) does not exist or is not writable to Composer."

just hit this again https://www.drupal.org/pift-ci-job/2586929 →

re-running tests made it pass

in this issue change drupalci.yml to run `WritableFileSystemValidatorTest` many times and see if it fails randomly

💯

@omkar.podey thanks I think that is about 5 fails out of 100 https://dispatcher.drupalci.org/job/drupal8_contrib_patches/143858/console

Not good 😔

It did fail randomly the error i can see is while $this->inspector->getConfig('minimum-stability', $dir); and some how we get PHP as a value.

Attached a screenshot below

Now i'm wondering if the changes i pulled from 8.x-2.x made it consistently pass somehow ?

This proves that with the changes from 8.x-2.x the tests are consistently passing, we can see different errors before and after merge (both cases without the check ($type === ProcessOutputCallbackInterface::OUT))

The conclusion for now,

1. The tests are passing consistently now after pulling the changes from 8.x-2.x
2. Leaving this issue open for now, if the inconsistencies surface again, I also saw some more issues but they seem to be fixed by the changes pulled from 8.x-2.x
3. Unassigning and setting to active

I got this error on #3343889-17: Drop support for end-of-life versions of Composer → on DrupalCI while I was not able to reproduce it on my local, re-running the test solved it for me.

Yep — all you can do for now is re-running it. That's why it's a random failure!

Per https://stackoverflow.com/a/57381161 this is perhaps happening because DrupalCI's /tmp is full? 🤔

Also: AFAICT this started happening since 📌 Improve test DX *and* confidence: stop using VFS Fixed .

Wim Leers → closed merge request !690

Closed the 8.x-2.x MR in favor of a new 3.0.x one. Moved all changes.

And then added more debug output.

00:02:15.868 php -r 'print sys_get_temp_dir() . "\n";'
00:02:15.945 /tmp

yet…

00:02:16.055 sys_temp_dir => no value => no value

Does that mean we can fix this by simply doing ini_set('sys_temp_dir', $some_directory_we_created)? 🤔

the problem might be with the permissions chmod 1777 the_tmp_directory might work

the open_base_dir is also empty open_basedir => no value => no value so just creating mkdir /var/www/tmp and changing permissions chmod 1777 /var/www/tmp maybe needs sudo ?

Retesting to see if it actually helped or was just a fluke

The tmp directory in not getting set

mkdir /var/www/tmp
15:15:58 chmod 1777 /var/www/tmp
15:15:58 php -r 'ini_set("sys_temp_dir", "/var/www/tmp");'
15:15:58 php -r 'print sys_get_temp_dir() . "\n";'
15:15:58 /tmp

This is now hypercritical. The failure frequency is higher. Probably because we run more composer commands now.

I guess it makes sense that calling ini_set('sys_temp_dir', …) does not help because the error message is occurring inside composer's \Composer\Console\Application::doRun():

            // Check system temp folder for usability as it can cause weird runtime issues otherwise
            Silencer::call(static function () use ($io): void {
                $tempfile = sys_get_temp_dir() . '/temp-' . md5(microtime());
                if (!(file_put_contents($tempfile, __FILE__) && (file_get_contents($tempfile) === __FILE__) && unlink($tempfile) && !file_exists($tempfile))) {
                    $io->writeError(sprintf('<error>PHP temp directory (%s) does not exist or is not writable to Composer. Set sys_temp_dir in your php.ini</error>', sys_get_temp_dir()));
                }
            });

— I bet that this is happening because we run tests with a high level of concurrency on DrupalCI, and that would make it _feasible_ that sys_get_temp_dir() . '/temp-' . md5(microtime()); runs at the exact same time in concurrent tests ⇒ a race condition!

👆 #28 means that #3345222 would help a LOT (but still would not solve it entirely): #3345222-4: Optimize Composer calls in Fixture Manipulator → .

🕵️‍♀️ The Composer code quoted in #28 was added in https://github.com/composer/composer/commit/28e9193e9ebde743c19f334a7294... almost 7 years ago. It was refined a few days later in https://github.com/composer/composer/commit/43eb471ec293822d377b618a4a14... and it's remained the same since then (other than non-semantical changes).

We could ask Composer to adopt hrtime() (TIL!) instead, that would make the chances of a collision far lower (microseconds vs nanoseconds).

But that will take time, which we don't have.

What if we can at runtime configure a unique temp dir? A quick experiment with a test.ini file like this:

[PHP]
sys_temp_dir = "/tmp/PID"

yields

$ php -c test.ini -r 'print sys_get_temp_dir() . "\n";'
/tmp/PID

🥳

Well I was very pleasantly surprised to discover \Drupal\package_manager\ProcessFactory — without that we might have had to make composer-stager modify its \PhpTuf\ComposerStager\Infrastructure\Service\ProcessRunner\AbstractRunner! 😅

Let's see how this fares… 🤓

While there is an ~80% failure rate today in kernel tests (in the sense that a test run is very likely to have at least one kernel test fail with the error in this issue's title), this one immediately passed … so … that suggests that #3 actually fixes this 😄

Queuing a second one to confirm… but I'm optimistic! 😄

I will land this without review from others because pretty much everyone reported being constantly blocked by this. Plus, the changes are very isolated, trivial to revert!

The last submitted patch, 33: 3338789-33.patch, failed testing. View results →

Yep, another all-green Kernel test suite run!

But … build & functional tests failed hard:

Exception: Warning: mkdir(): File exists
Drupal\package_manager\ProcessFactory->create()() (Line: 79)

I think that's easy to fix though 🤞

Due to the agreed hypercriticality (!) of this issue, I'm okay merging whatever fixes it post-haste without review. Consider this a "blanket RTBC".

That did it! 🥳

Both green, committed! 🚢

Wim Leers → closed merge request !745

Now filed upstream PR with Composer to hopefully get it fixed there: https://github.com/composer/composer/pull/11375 🤞

Automatically closed - issue fixed for 2 weeks with no activity.