D7 contrib pipeline tests fail when requiring a 3rd-party contrib module

Note that I’ve run into this same situation when integrating GitLab CI for contrib D7 and D10 and core D7 and D10.

A few core issues were actually found, also in the contrib modules that were tested, and of course, in the gitlab templates.

So keep an open mind about where the actual error could be and why it’s happening.

DrupalCI configuration is different from the one in the gitlabci templates.

As long as we don't have more specific information on where the error may be in this project, I'm going to postpone this issue.

I have made some progress on tracking down the problem. As I cannot replicate it locally, I am patching Core with debug in scheduler/-/merge_requests/99. The thing that finally causes the error is that permissions defined by the contrib module are not always available, hence the createUser fails. It seems that on the failing tests module_implements('permission') does not return the full list of modules implementing hook_permission. On the tests that pass, the list is longer, and includes 'scheduler'. I am continuing to track backwards to try to find the cause.

I realised that the Rules test without the 'rules' module runs the same sequence as the main Scheduler test. The permissions are ok. This is the same behaviour as the Date tests. Therefore it seemed it is not a problem with the Date module doing anything wrong, it is more likely some problem caused when a new 3rd-party module is installed from the setUp().

My debugging led right back to /includes/module.inc module_enable() The list of modules to enable is checked against and 'date' and 'rules' modules are not found, so the process just stops. Hence not even Scheduler is enabled, this is why Scheduler's permissions were available but after a reset they are not.

The additional modules are loaded in the composer job via the project's composer.json. But they are put in $CI_PROJECT_DIR/vendor/drupal. This is not a location that Drupal 7 uses for contrib modules, they need to be in $CI_PROJECT_DIR/$_WEB_ROOT/sites/all/modules

I have a working proof of a solution in Scheduler MR105 by adding the following to the gitlab template

composer:
  after_script:
    - ls -lG $CI_PROJECT_DIR/$_WEB_ROOT/sites/all/modules
    - ls -lG $CI_PROJECT_DIR/$_WEB_ROOT/sites/all/modules/custom
    - ls -lG $CI_PROJECT_DIR/vendor/drupal
    # Third-party modules have been loaded via the projects composer.json into
    # vendor/drupal but they need to be moved into sites/all/modules for D7.
    - cp -LR $CI_PROJECT_DIR/vendor/drupal/* $CI_PROJECT_DIR/$_WEB_ROOT/sites/all/modules
    # The directory vendor/drupal/drupal has been copied but must be removed, as this
    # is a duplicate and causes problems.
    - rm -R $CI_PROJECT_DIR/$_WEB_ROOT/sites/all/modules/drupal
    - ls -lG $CI_PROJECT_DIR/$_WEB_ROOT/sites/all/modules

The results of the above can be seen in https://git.drupalcode.org/project/scheduler/-/jobs/172833

I expect there is a better way to install the 3rd-party modules directly into $CI_PROJECT_DIR/$_WEB_ROOT/sites/all/modules and also to avoid having to delete the /drupal/drupal directory.

MR59 ready for review, or rather, ready for comments. It includes listing the directories for info, which will probably be removed from the final version. Other things to discuss

1. Is there a way to get the modules loaded directly to the correct directory in the first place?
2. If not, then should the vendor/drupal/* directories be moved, instead of copied?
3. Is there a better way to copy the vendor/drupal/drupal directory. Something like piping the list, grep or sed to remove 'drupal', then do the move or copy

Could this be due to having "drupal/composer" added?
https://git.drupalcode.org/project/gitlab_templates/-/blob/1.0.x/include...

This might not be needed for the vast majority of contrib modules so I'm actually thinking we should remove it or put it behind a variable so that modules go to the right place by default.

From community slack: https://drupal.slack.com/archives/CGKLP028K/p1697539018203129

@fjgarlin
I got GitLab CI working on project_composer module, but had to override the template script. The module requires composer which conflicted with drupal/composer. I couldn’t find where drupal/composer was actually used, so I removed it, and it works https://git.drupalcode.org/project/project_composer/-/commit/e949cf78b1e....

These three projects have different composer needs.
Projects to try:
* `api` D7
* `scheduler` D7
* `project_composer` D7

_D7_DRUPAL_COMPOSER_NEEDED is set to 0 by default and should ideally work for at least "scheduler" and "project_composer". When set to 1 it should work for "api".

To test this in each project:

  - project: issue/gitlab_templates-3392363
    ref: 3392363-fix-modules-directory

"project_composer" works with these changes: https://git.drupalcode.org/issue/project_composer-3134858/-/pipelines/39429

Unfortunately it does not work with Scheduler when _D7_DRUPAL_COMPOSER_NEEDED is set to 0
See https://git.drupalcode.org/project/scheduler/-/pipelines/41591 which is a plain branch pipeline with the added variables for REPO, REF and SHOW_ENVIRONMENT.

The required 3rd-party modules (drupal/date and drupal/rules) retrieved, but I suspect that they are still going into $CI_PROJECT_DIR/vendor/drupal. You have removed the debug ls so I can't be sure. But if they are there, they won't be moved out to sites/all/modules because that code is now only done inside the conditional block.

Thanks for the test. I might end up an MR on the "scheduler" project so I can play with it too.
We might need to add more tweaks inside the if and also some defaults outside of it too.

I am going to commit Scheduler 7.x MR105 which has the composer: after_script: processing. This is so that I can get 7.x branch tests passing, and will remove that code when this issue is fixed. I need to make a final 7.x release soon, so want to see the test pipeline passing.

Current status.

Works:
* project_composer: https://git.drupalcode.org/issue/project_composer-3134858/-/pipelines/39429
* api: https://git.drupalcode.org/project/api/-/jobs/335421

Does not work:
* scheduler: WIP

Update: all three modules that are being tested get green results. Some need the new variable, some don't, and that's a good thing, because it's the goal of this change.

Given the results reported in #14, where all three D7 modules are now running green with the changes in this issue, I'm going to put it as "Need review".

This looks good. So, there are two differences between when I tested in #13 (Nov 2) which failed, and the latest runs which pass:

1. You have reinstated vendor/bin/drush --root=$_WEB_ROOT en -y $MODULE_NAME which had been commented out
2. Scheduler has to have _D7_DRUPAL_COMPOSER_NEEDED: 1 because that is where the 3rd-party modules get moved into the correct folder

So we are still getting the 3rd-party modules loaded into vendor/drupal and we have to copy them into sites/all/modules. Is copying the correct thing to do? We could make symlinks for them. Or move instead of copy. Just asking ...

I'm fine with either. Currently, we are copying and then removing the non-valid one, so that might be simplified with a "mv" command instead.

Yes I coded it the simple way first :-) Do you think that duplicating the modules could somehow lead to a problem in future? (I don't know exactly what, but it feels wrong to have two copies). But how do we change cp -LR $CI_PROJECT_DIR/vendor/drupal/* into a mv that move vendor/drupal/drupal ? Or maybe we move all, then move that one back again?

We can do an extra “rm” command after this line https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/59/...

From what you are saying above, I infer that we don't need to leave the vendor/drupal/drupal there, so I have tried with a

It also seems like the "drupal/composer" needs to be in the "vendor" folder... not 100% sure of it, but it's defo not a normal drupal module (no info file, no module file...).

So I guess we can make an exception for these two, or maybe just revert to the "cp" line. If the tests are not complaining, then maybe we can live with having different copies. Maybe if some users/modules report a problem with this we can have a deeper look, but for now (and for D7) it's good enough. I think this feature will defo make D7 adoption easier.

This version moves the folders but leaves 'drupal' and 'coder' untouched. The composer directory is already moved to vendor/drush/drush/commands/composer (you did that work before this).

The ls | grep | xargs works locally, hope it does the same here. I have included some debug ls listings to show what was done.

Scheduler passes again with this latest change
https://git.drupalcode.org/project/scheduler/-/pipelines/50765

I will test the other two projects also.

Here's the confirmation

It does not appear to work here: https://git.drupalcode.org/project/api/-/jobs/345889

The output for composer require --dev drupal/composer:1.x-dev is there and is correct. but no further output and it does not get to php symlnk_project.php

I've tested the mv locally (just in terminal, not running the actually gitlab pipeline). There is no problem when there are no directories to move. If the source /vendor/drupal does not exist I get "No such file or directory", but we don't see that in the log here. Adding debug before the mv.

The "ls" commands run, we can see that there are no folders in the second one and only the "coder" and "drupal" in the first one. The output is here: https://git.drupalcode.org/project/api/-/jobs/346610

And then it fails, so it's still that piped command. Assuming that it's due to the empty results.

If you're trying locally on a Mac, it won't fail (I am and it didn't...).
Added a flag as reported here: https://stackoverflow.com/questions/8296710/how-to-ignore-xargs-commands...

If you're trying locally on a Mac, it won't fail

Yes I am. Thanks for finding that flag.

It wasn't even making it to the "mv" part. See this: https://git.drupalcode.org/project/api/-/jobs/346892
ls $CI_PROJECT_DIR/vendor/drupal | grep -Ev '(drupal|coder)' was returning 1 already.

https://stackoverflow.com/questions/42251386/the-return-code-from-grep-i...

Working on a fix.

Back to working:
* api: https://git.drupalcode.org/project/api/-/pipelines/50955
* project_composer: https://git.drupalcode.org/issue/project_composer-3401810/-/pipelines/50962

Not working:
* scheduler: https://git.drupalcode.org/issue/scheduler-3401828/-/pipelines/50961 (not sure why or how is connected here)

I got the exit_code check to work, but it then failed with
args: invalid option -- 'S'
so I removed that option which you added here. Was there a failure that needed your change? My local xargs man shows

 -S replsize
             Specify the amount of space (in bytes) that -I can use for replacements.  The default for replsize is 255.

Happy for it to be removed, I was just trying things that may or may not make a difference. Good progress!!

Scheduler passes now https://git.drupalcode.org/project/scheduler/-/pipelines/51084
Need to check the other projects.

Composer jobs:
- GREEN https://git.drupalcode.org/issue/scheduler-3401828/-/jobs/349293
- GREEN https://git.drupalcode.org/issue/project_composer-3401810/-/jobs/349299
- GREEN https://git.drupalcode.org/project/api/-/jobs/349309

The pipelines for those jobs are also green :-)

This is excellent. It's ready, apart from one comment I requested.

Interesting to note that we were sooo close before. Here's the sum total of the last 23 commits, it only required || true to be added!

I know, that uppercase TRUE threw us off into a spiral of commits.
I just added the comment you requested.

Thanks for adding the comment. Happy for this to be RTBC is you are.

Great that we had three D7 modules to test against, all with differing requirements

• composer 3rd-party modules
• composer 3rd-party modules
• no composer required

Awesome. I'll be committing this Monday morning so I can follow up with this and the adaptations needed for the projects we've been testing on.

Thanks so much for the back and forth and for the GREAT collaboration here.

Merged and marked this as fixed. This was a big one for D7 contrib. Thanks so much for your help here @jonathan1055, really appreciated!

You're welcome. Great to be working in a team, co-ordinating when one of us is working on it, and the other is not.

Automatically closed - issue fixed for 2 weeks with no activity.