Ensure that intentionally failed test stage directories are deleted after tests complete

Created on 18 January 2023, almost 2 years ago
Updated 19 April 2023, over 1 year ago

Problem/Motivation

Realized that 🐛 GitExcluder should not ignore .git directories that belong to packages installed by Composer Fixed 's new test additions are leaving behind stage directories in the staging root. It's very unlikely to result in collisions. But it can accumulate quite a bit of disk space in principle.

Let's clean up behind us.

Until 📌 Improve test DX *and* confidence: stop using VFS Fixed this wasn't necessary, because the staging root was always inside VFS, but #3328234 will change that.

Steps to reproduce

See above.

Proposed resolution

TBD

Remaining tasks

TBD

User interface changes

None.

API changes

None.

Data model changes

None.

📌 Task
Status

Fixed

Version

3.0

Component

Code

Created by

🇧🇪Belgium wim leers Ghent 🇧🇪🇪🇺

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇧🇪Belgium wim leers Ghent 🇧🇪🇪🇺

    🐛 GitExcluder should not ignore .git directories that belong to packages installed by Composer Fixed is postponed again on another issue, so matching that increase in blockers.

  • Status changed to Active almost 2 years ago
  • 🇧🇪Belgium wim leers Ghent 🇧🇪🇪🇺
  • 🇺🇸United States tedbow Ithaca, NY, USA
  • Assigned to yash.rode
  • Status changed to Needs work over 1 year ago
  • 🇮🇳India yash.rode pune

    Before starting with this I have a thought about how can we achieve this,
    one way is to check each and every test and destroy the stage where it is not destroyed already.
    the other way can be using tearDown() but I don't know how can we get the stage in that case?

  • 🇮🇳India yash.rode pune

    As discussed in meet one ways to do this are,

    1. In assertResults() we can add a finally block and delete stage in that
    2. In teardown we can delete the stage directory
  • 🇺🇸United States tedbow Ithaca, NY, USA

    #12
    I think we want to go with 2)

    but slightly different in tearDown() we should delete `\Drupal\package_manager\PathLocator::getStagingRoot()`
    and \Drupal\package_manager\PathLocator::getProjectRoot() but we should only delete getProjectRoot() if is directory that is under `DrupalFileSystem::getOsTemporaryDirectory()` which will be the case if it is created in `\Drupal\Tests\package_manager\Kernel\PackageManagerKernelTestBase::createTestProject`

  • 🇺🇸United States phenaproxima Massachusetts

    and \Drupal\package_manager\PathLocator::getProjectRoot() but we should only delete getProjectRoot() if is directory that is under `DrupalFileSystem::getOsTemporaryDirectory()` which will be the case if it is created in `\Drupal\Tests\package_manager\Kernel

    I think we could probably simplify this; if the project root is not the same as $this->getDrupalRoot(), we can delete it. I think this is the case because, in a core clone, $this->getDrupalRoot() is what the path locator will report as the project root, by default, unless we specifically override it.

  • @yashrode opened merge request.
  • Assigned to tedbow
  • 🇮🇳India yash.rode pune

    Need help with the test failure, \Drupal\Tests\package_manager\Kernel\SymlinkValidatorTest::testSymlinkToDirectory is failing with rmdir(/private/tmp/package_manager_testing_roottest95214460/active): Directory not empty, is it because we are trying to delete symlink? or something else?

  • Issue was unassigned.
  • Status changed to Needs review over 1 year ago
  • 🇮🇳India yash.rode pune

    Right now I am skipping \Drupal\Tests\package_manager\Kernel\StageOwnershipTest::testStageDestroyedWithFileSystemError because we are overriding chmod(), unlink(), rmdir() which we need in tearDown(), is there a way to not skip this test?

  • Status changed to Needs work over 1 year ago
  • 🇺🇸United States phenaproxima Massachusetts

    We shouldn't use the Drupal file_system service to do the delete, precisely because, in some situations, we override and mock it. Instead, we should use Symfony's Filesystem class, which has a remove() method which can do this for us cleanly.

  • Status changed to Needs review over 1 year ago
  • Assigned to yash.rode
  • Status changed to Needs work over 1 year ago
  • 🇺🇸United States phenaproxima Massachusetts

    This looks straightforward but there are some things there that I don't understand why they're necessary...

  • Issue was unassigned.
  • Status changed to Needs review over 1 year ago
  • 🇮🇳India yash.rode pune

    Answered questions and removed the unlink() calls.

  • Assigned to phenaproxima
  • Status changed to Needs work over 1 year ago
  • 🇺🇸United States phenaproxima Massachusetts

    I don't think we want to be modifying permissions of the system-wide temp directory.

    If NOT doing that causes some tests to fail, I'd say that points to a flaw in those tests. I will take this on to figure out why we'd need to do that, and fix the affected tests.

  • 🇮🇳India yash.rode pune

    I am not able to post comment on GitLab, this is not true I guess, example$staging_root will be /private/tmp/package_manager_testing_roottest30218963/stage so it's parent won't be temp directory.

  • Assigned to tedbow
  • Status changed to Needs review over 1 year ago
  • 🇺🇸United States phenaproxima Massachusetts

    Assigning to @tedbow for final review.

  • Assigned to phenaproxima
  • Status changed to RTBC over 1 year ago
  • 🇺🇸United States tedbow Ithaca, NY, USA

    This looks good. I pushed up 1 commit because there was 1 folder not being deleted. @phenaproxima if this looks good and tests pass, feel free to commit.

  • 🇺🇸United States phenaproxima Massachusetts
  • Status changed to Fixed over 1 year ago
  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024