Integrate Symfony Console component to natively support command line operations

Here's a proof of concept.

1. Apply the patch.

2. Run composer update

Then you'll be able to do this:

core/console

This will show you a help screen.

I've added three commands: cache:clear, module:install, and module:uninstall.

You can say core/console help module:install and such to get help with individual commands.

All three work, though uninstall managed to bork my Drupal when I told it to uninstall node. Obviously this is proof of concept, not production code.

Teensy bit of refactoring, added user:login, which returns a one-time login URL for the site.

Because it makes no sense whatsoever to be able to type core/console user:login 1 | pbcopy, does it? :-)

The issue summary should really describe at which point drush starts and when the console ends.

@mile23
Once we reach an agreement that we ant to have the symfony console in core, we should open on issue to pull in the code.

@dawehner: It wouldn't be the symfony console in core. It would be the Drupal console. :-)

Let's add console-based objects to core, which Drush can then use.

How about replacing core/scripts/drupal.sh?

core/console drupal:request /path will give you the HTML for that path.

It has an option to return only the response code, like this:

$ core/console drupal:request /nonexistent -s
404

+1,000.

A key aspect here is a pluggable architecture.

First step towards that is to move e.g. the cache command into Drupal\Core\Cache\Console\ClearCache.

In other words, in abstract: <namespace>\Console\<Command>.php

From there on, things will become much more clear.

@sun: Exactly the conversation I was hoping for. :-)

I think the big deal here is that any command should be very small in terms of lines of code. Let's say 20 for the execute() method. I think setting an artificial (but reasonable) limit on the size of any command implementation forces people to think about how to refactor and improve the code they're driving, rather than add to the console complexity.

For instance: We'd have one Command base class that boots Drupal for you (see above), and if you discover that you need to change the way it does that, or you need to boot Drupal in a different way, then really the solution to that problem should move 'upstream' to Core or Component namespaces to better deal with your use-case.

This is one reason I think we should have a Drupal\Console namespace, with Core\ and Component\ mirrored underneath it, similar to the way it works for PHPUnit's tests/ directory.

@Mile23, @sun & @dawehner We have a project named console => https://drupal.org/project/console

The purpose of this project is to leverage the Symfony Console Component to provide a CLI tool to automate the creation of drupal 8 modules by generating the directory structure for a module, routing, controllers, forms, services, plugins and required configuration files.

We have the Symfony Console component fully functional

This will be the installation instructions for the module

$ cd path/to/drupal8.dev
$ COMPOSER_BIN_DIR=bin php composer.phar require --dev drupal/console:dev-master
$ ./bin/console --help

After installation you can access the console at

$ bin/console

Related to the scaffolding commands we have:

Module generator
$ bin/console generate:module

Controller generator
$ bin/console generate:controller

Form generator
$ bin/console generate:form

Plugin Block generator
$ bin/console generate:plugin:block

Also makes the automatic registration of commands for each installed module.

Your commands must inherit from the class
Symfony\Component\Console\Command\Command

True @dmouse I forgot the automatic registration of command part.

Also after a talk with @Crell at a Camp he recommended to decouple the console component and the scaffolding tool and it sound kinda interesting this could be an opportunity to do what he mentioned

Hey cool. Looks a little like this: https://drupal.org/sandbox/mile23/2230027

:-)

The autodiscovery part is great. I agree about decoupling. If you check the patches above you see you can have a light touch.

re: #7: @Mile23:

I cannot agree with the structure you're suggesting. Every component is supposed to be an independent, decoupled component. Components MAY ship with Console commands. If they do, then the code is provided, owned, and maintained by the component maintainers.

The only reason for why tests are split into an own namespace is that all test code should not be available and loadable at regular runtime.

Every component and every module can have Console commands. Just like they can provide e.g. Entity type plugins. Therefore, the proper home for Console commands is in a relative ./Console subnamespace/subdirectory of each component/module.

There is no need to re-invent this pattern; simply have a look at how Symfony bundles and many other PHP components (that are only using Symfony Console) are doing it. What you'll see is a ./Console subnamespace/subdirectory.

That structure allows for clean discovery of available Console commands. The defined autoload namespaces are driving the discovery. So when e.g. Guzzle ships with Console commands, then those will be automatically discovered and offered, too.

If the Console component doesn't ship with a namespace-based discovery already, then we already have plenty of our own. Just use 'em.

This is how it's supposed to work:

https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Htt...

The only addition/customization to that is that our components in Drupal\Core\* do not follow the concept of Bundles (only Drupal\Core itself is an equivalent), but as mentioned before, our plugin discovery actively supports to treat every core component as a separate unit, and we should continue to do that.

Plugin discovery is an important keyword: Console commands are pretty much the textbook example of plugins.

Plugin discovery is an important keyword: Console commands are pretty much the textbook example of plugins.

Of course we want to autodiscover console commands. See this from the patch in #5:

+++ b/core/console
@@ -0,0 +1,24 @@
+// @todo: Replace this with autodiscovery.

However: They're not plugins. They don't even need a booted Drupal. There's nothing for them to plug into. It's like saying PHPUnit tests are plugins. Even under Symfony, commands aren't plugins.

Note that in Symfony2, they dropped the term 'plugin,' and rightly so. It's a confusing term. Guess what replaced it?

Hmmmm.... You've linked to code that uses Symfony\Component\Finder\Finder. Sure would be handy to have that, too, for autodiscovery. :-)

@Mile23 in the console project this is the discovery and registration commands

https://github.com/hechoendrupal/DrupalAppConsole/blob/master/lib/Drupal...

Thanks.

Currently traveling, so this will go slowly.

Very work-in-progress-y.

Adds some utility classes for finding stuff in core, which might be a bit out of scope but we roll with it. The challenge is to find commands before bootstrapping Drupal.

Also gives a basic framework of stuff to debate. :-)

To install, you need to run composer update. Then you can do core/console to see the help screen, and core/console drupal:cron to see the autodiscovered cron command in action.

1. +++ b/composer.json
   @@ -23,12 +23,16 @@
   +    "symfony/console": "2.4.*",
   

   this dependency shouldn't be ~2.4 instead of 2.4.*?

2. +++ b/core/lib/Drupal/Console/Command/CommandBootstrapBase.php
   @@ -0,0 +1,73 @@
   +    require_once __DIR__ . '/../../../../includes/bootstrap.inc';
   

   i thought the drupal api already autoloaded all that needs to be autoloaded?

3. +++ b/core/lib/Drupal/Console/Command/CommandBootstrapBase.php
   @@ -0,0 +1,73 @@
   +      $container = $kernel->getContainer();
   ...
   +      $container->get('request_stack')->push($request);
   

   have you checked how app/console container:debug does it? i think if the container is dumped there may be no need for this

4. +++ b/core/lib/Drupal/Console/DrupalApplication.php
   @@ -0,0 +1,72 @@
   +  private function discoverCommands() {
   

   can we totally avoid this type of stuff and just $this->add(new CommandHere) ?

   or maybe instead of having this let's tag the service commands definitions and let the compiler pass gather the right commands needed. Having this instead is really ugly and prone to all sort of messiness.

5. +++ b/core/lib/Drupal/Core/Discovery/DrupalFinder.php
   @@ -0,0 +1,94 @@
   +      $drupal_root = realpath(dirname(dirname(dirname(dirname(dirname(dirname(__FILE__)))))));
   

   please do not repeat symfony's way of doing this if we can register the command as services

Thanks, @cordoval.

Point 1: I tried various combinations of ~ and .* and others too, and this was the best result. See also the 'prefer-stable: true' line.

Point 2: The console application (\Drupal\Console\DrupalApplication) only has the composer-generated loader at that point. This is intentional... Only bootstrap if the command we're running needs it.

Point 3: These are based on the current front controller code, ripped from drupal_handle_request(). Once the kernel is refactored, this will all change. That's one reason it's abstracted the way it is. Will poke through container:debug, but really what to do next depends on 📌 Refactor bootstrap to better utilize the kernel Fixed

Point 4: See discussion above about plugins and autodiscovery. :-) In my mind, we want to include commands from the standard contrib places, as well as from /vendor. (Allowing composer require somevendor/someconsolecommand is very appealing to me.)

Point 5: I want it to be true that if you just type core/console you don't bootstrap a Drupal, and having a container or a kernel is not a requirement of any command you might write. It'd be great to leverage what's already there, but I'm not sure that Drupal's container can be loaded or updated without the database or without bootstrapping a bunch of stuff at some level. That is: I really don't know. :-) Any pointers would be helpful.

Here's some development on the console. Important changes include:

The console app and all commands are services, discovered from yaml files.

The yaml files are parallel to the standard Drupal ones. There is a core.console.services.yml file, and the discovery process finds files named *.console.services.yml.

This follows the pattern of the rest of Drupal, but is in parallel because the core services have lots of dependencies that they really shouldn't. :-)

Woops... Wrong diff. :-)

Re-roll.

1. There does not seem to be a generic way to expose Symfony Console commands. Symfony's FrameworkBundle uses a common discovery method, but that relies on bundles, which are a concept that does not exist within Drupal.
2. Do we have good use cases for using a console-specific container?
3. Why do we register the console application as a service in the console-specific container instead of building it in the console file? Testability?
4. Is there anything else we need to do in order to provide a good basis for this to replace Drush?

1) :-)

It's true that there isn't a generic way to do discovery. This is one of the central problems of this exercise... It would be great to make commands into services, so they could be discovered by and cached by the container, but the container has too many dependencies (and is a confusing mess to try to build outside the kernel). I also tried using symfony/finder to do a file search and then determine whether the file is a subclass of the command base class. Fun times.

2) The use-case for the console-specific container is that I wanted to figure out how to make commands into services. The problem of dependencies led me to have a parallel system of service definitions. The idea was to eventually fold those in with core as was possible.

I'm currently working on having some commands be hard-wired in to the app, just adding them in the constructor of the app. This is for commands that need to show up very early in the boot process, like drupal:install.

In an ideal world, there would be a policy document on what dependencies a service can have. There's also this: #1973618: DIC: Lazy instantiation of service dependencies (ProxyManager for "proxy services") →

3) The console app is a service because this allows us to add a compile phase to the container. It finds tagged commands and adds them to the app automagically. Also, like you say, testability. Symfony/console has its own testing infrastructure.

4) No desire to replace drush. :-) I simply want to replace /core/scripts/ with testable code. Drush can do complex things that the Drupal console shouldn't. Over time, it might be that people make drush-equivalent stuff under this system, but that's out of scope here.

Hi, can we talk on IRC, just to hash a few things out?

This patch includes the vendor packages. I did not make any other changes yet.

The last submitted patch, 25: drupal_2242947_25.patch, failed testing.

There is a binary file in here, so use --binary when creating patches.

I cleaned up the directory structure and code comments. I also added a cache clear command and tests.

I'd like to add at least one command that belongs to a module so we know the services finder does not only *.console.services.yml for core, but for modules as well. What about the user cancellation command, which belongs to Drush, but is fairly simple? The only module-specific command in core is run-tests.sh, which is too big for this issue to port to a command.

The last submitted patch, 28: drupal_2242947_28.patch, failed testing.

The last submitted patch, 30: drupal_2242947_30.patch, failed testing.

We don't need to add a module to test file discovery. In subsequent versions I added vfsStream to test that. https://github.com/mikey179/vfsStream/wiki

The earliest patches in this issue have a few different commands that could be re-added. I took them out to concentrate on the framework rather than the application.

Will do more review tonight.

Thanks. :-)

Should fix the failing PHPUnit test

The last submitted patch, 34: console-token-2289447.5add5f2.patch, failed testing.

missed the --binary

But besides that, it doesn't work.

After fixing the include path for bootstrap.inc (needed another ..) I get

Getting 'PHP Fatal error: Class 'Drupal\block\EventSubscriber\CurrentUserContext' not found'

Traced that back to #2278541: Refactor block visibility to use condition plugins → .

But essentially any module-based event-listener would cause this to blow up because the call to \Drupal::config() in drupal_bootstrap_pagecache() triggers a build before modules are registered with the autoloader.

The issue was #2228215: Remove module check in DrupalKernel → , we need to prevent loading.

Core only changes

+++ b/core/lib/Drupal/Core/Console/ServicesFinder.php
@@ -0,0 +1,24 @@
\ No newline at end of file

Only issue I can find :D. Great work everyone.

I have two main concerns:

1. Is it guaranteed that drush is going to use this? We don't want two approaches, one living in core (the current scripts don't count as 'two approaches', I'd happily commit an issue that is just git rm for those).

2. Is there any evidence that the console component has been used for tasks such as migrating tens/hundreds of millions of items over several hours. We know drush can do this, and it took some time to get that right in terms of memory management, starting new processes etc.

@catch:

1) It'd be great to get the opinion of one of the Drush leads. They should walk themselves through the 'your first console app' documentation here: http://symfony.com/doc/current/components/console/introduction.html

That said, however: Drush can do big complicated stuff like site aliases for three major versions of Drupal. The Drupal console is for smaller, easily implemented stuff like clearing the cache. Also it will be tested in core's continuous integration, so you'll always be able to clear your cache when you install Drupal.

2) Symfony/console is a framework that handles the text input and output parts of making a console application. The rest is up to us.

I'd also like to point out that the extent to which this is easy is a consequence of good OOP in Drupal. The better the refactoring, the easier it is to make a console command for it, and also the more maintainable that command is.

For instance, this from @larowlan above:

But essentially any module-based event-listener would cause this to blow up because the call to \Drupal::config() in drupal_bootstrap_pagecache() triggers a build before modules are registered with the autoloader.

Oops. Dependency problems in the config system... Looks like something to fix in core. :-)

jibran → queued 37: symfony-console-2242947.6cb8b52.patch for re-testing.

The last submitted patch, 37: symfony-console-2242947.6cb8b52.patch, failed testing.

This feels like 8.1.x material at this point, I think.

@Mile23, you should take a look at this issue in the Drush issue queue Leverage symfony/console package, if you haven't already seen it. I do like the idea of a CLI that works with core without users having to do anything else, but as others have noted, Drush might be the better place for this.

Again: This is not an issue that says 'replace Drush with a native console.'

It is an issue to add functionality to Drupal that everyone wants, which is why Drush exists. If memory serves, Drush came about because of the same sort of friction that's facing this issue. Only, ironically, now it's because there's a Drush that there's friction against core console here.

Some of the problems solved by adding a console component are:

1) Functionality that is required for core is maintained along with core. For instance, a command-line installer will be maintained and will work with core always, because we have CI and it's not a separate project.

2) The Drupal console will be able to find commands within vendor/. You can add a command, put it on packagist, and everyone can get it.

3) I won't have to wonder whether the current Drush will save me time or waste it for simple tasks. This is due to #1 above.

4) You really should take a look at https://www.drupal.org/project/console → which generates Drupal 8 modules for you. Just like Symfony does for all the Symfony people. Very convenient. Would be super useful to have in core, or as a contrib project which you add through composer.

That's the sort of thing we'd enable here. Code's ready, FTMP, because it's not really that complex. Drush has to work with a wide array of Drupal versions, whereas this project only cares about 8.

The decision is whether to support CLI for common tasks such as installation. I'd write the installer, but I'm not sure why I'd do it if no one cares.

Adding in 8.1.x is fine.

Here's a re-roll.

The last submitted patch, 46: 2242947_46.patch, failed testing.

Reroll against 8.0.x, so I'm setting the version. Will set it back.

updating the issue summary with the 8.1.x rationale

Yah, I set it to 8.0.x for the testbot, because 8.1.x branch is way behind.

The last submitted patch, 50: 2242947_50.patch, failed testing.

Looking forward to this

Adding this as a related issue: #2493807: Add symfony/console component to core →

That issue only adds the component to vendor/ and doesn't do any integration.

Just as a reminder, for when this problem is changed to active status again, we have been working on this contributed project http://drupalconsole.com/ there is already code that could be reused on core or contributed space to facilitate the use of Console component in order to enable Drupal users to create commands easily.

What's the status of this issue after #2493807: Add symfony/console component to core → ? Is this a task now? Is this still pending till 8.1.x?

Yes, because it's still a feature request. Console was committed only in support of closing the critical issue at #2447573: [meta] Make sure 8.x - 8.x hook_update_N() testing is possible → .

1. #2447573: [meta] Make sure 8.x - 8.x hook_update_N() testing is possible → is a critical issue.
2. It requires the current shell script to be converted to a console command for reasons related to QA.
3. To do this, Symfony's Console component was added to Drupal core in #2493807: Add symfony/console component to core → .
4. Any Console command requires a console application. Such applications can handle multiple commands, and are not tied to any specific one of them.
5. As such, it makes sense to add the application with command discovery in a separate issue, such as the one you are currently reading, and then use #2447573: [meta] Make sure 8.x - 8.x hook_update_N() testing is possible → to add specific command for database dumps.

Following this reasoning, this issue is actually a blocker of a critical issue and should therefore be made active and reset to 8.0.x again.
Building the application does not mean all shell scripts should or can be converted to console commands for 8.0.x. It merely means the basis is in place to do this anytime we think is appropriate, such as for 8.1.x or later minor versions.

Any Console command requires a console application. Such applications can handle multiple commands, and are not tied to any specific one of them.

While this is the default behavior, it is possible to use the console component to http://symfony.com/doc/current/components/console/single_command_tool.html">support single commands. In the interest of moving along the upgrade path, I'd prefer to save the generalized application with multiple commands for this issue/8.1.

No, that critical will just use a simple application for that one command in core/scripts/some-app.sh, which is three lines of code.

My favorite example is still:

https://github.com/msonnabaum/XHProfCLI/blob/master/src/XHProfCLI/App.php

Clear, easy and concise.

#61 I totally agree with @Xano, there is a need for the Application class, we have been discussing this on the IRC earlier today.

#62 I like the idea of @Fabianx, we can create a simple Application Class to fix this issue.

#63 @jhedstrom The example you provided is a single command application but it also requires two classes, the Application class MyApplication and the Commands class MyCommand

@jhedstrom: Thanks for the example. I see your point and agree. I would like to ask we focus on building the small application in such a way that it will only support the single command, but is not specific to it, meaning that in 8.1.0 we can extend it to support multiple commands without having to break BC.

Still no clear for me why we keep talking about supporting a single command when we are trying to convert multiple scripts to console commands https://www.drupal.org/node/2289405 →

If we code the Application class (a small and simple one), adding more commands does not look like a big problem.

 $this->addCommands([
      new PasswordHashCommand(),
      new RebuildTokenCalculatorCommand()
 ]);

No, there will be one app per command for now and a shell script to wrap it.

Conversions won't happen until 8.1 - per this being a feature request.

You can play all you want for 8.1, but lets get D8 shipped first, please.

*This* issue represents an integration which does the following things, among others:

• Performs discovery of commands in Drupal-contrib and composer-based packages.
• Allows the console to use the container, and thus the kernel.
• The console builds its own service container to accomplish discovery, but this implementation detail would probably change.

These are features which would be nice to have, but aren't vital at this late stage in the 8.0 release cycle.

Scripts for 8.0.x should use the one-file-per-script model that console allows through fluent interfaces and closures, so that we get:

• The benefit of the framework.
• The ability to refactor these scripts to an integration more easily should the need arise.
• Console's testing framework.

Also, #2289405: [META] Port all shell scripts to console commands → is postponed on this one. It would be great to refactor all of them for grins, but not during beta.

No, there will be one app per command for now and a shell script to wrap it.

If so, could you please explain the need to enforce one app per command?

could you please explain the need to enforce one app per command?

In order to limit the disruption during beta.

Just did a re-roll to see how things have changed.

The commands work (as PoC), the tests fail, and you'll need to do composer install to get symfony/finder.

This is still postponed for 8.1.x, but the patch applies against 8.0.x.

In order to limit the disruption during beta.

How is this a need right now? Core does not ship with any Symfony console commands right now, so adding any command will most likely not be disruptive, but additive. On top of that I have seen no reason not to ship the first command with a generic console executable/application that can discover other commands in the future, so we'll at least have a single executable. Commands themselves would then be responsible for booting core, or not, for instance.

I have seen no reason not to ship the first command with a generic console executable/application that can discover other commands in the future, so we'll at least have a single executable. Commands themselves would then be responsible for booting core, or not, for instance.

Agreed.

However, if this patch is postponed until 8.1.x, then it's not happening, and we should limit the footprint of any console-based scripts so we don't end up with a bunch of DrupalWTF later.

So change the status here and make the case if you'd like.

Also: It sure would be nice if it were easier to boot core in one or two lines of code, wouldn't it? :-)

Since the 8.1.x branch has been merged, here's a reroll/fix patch against it.

This patch adds the symfony/finder component to core, so there is a big patch and a core-only patch.

1. +++ b/core/composer.json
   @@ -9,6 +9,7 @@
   +    "symfony/finder": "2.7.*",
   

   Do we need to add symfony/finder here? I know the Behat project removed use of this due to memory issues, and now uses native php instead.

2. +++ b/core/core.console.services.yml
   @@ -0,0 +1,24 @@
   +services:     ¶
   

   White-space.

If this were in, would it allow us to start refactoring the various scripts in code/scripts to use console? The one script currently using console is dump-database-d8-mysql.php, but it relies on a single-command application class (DbDumpApplication). Ideally we'd have a more generic single-command class that these scripts could use.

We would indeed be able to do that. With generic command discovery (plugins?), every module (core & contrib) can provide commands.

If this were in, would it allow us to start refactoring the various scripts in code/scripts to use console?

Convert existing core/scripts/ to Console you say? :-) #2624926: Refactor run-tests.sh for Console component. →

Basically what this integration gives us is a discovery mechanism and a unified UX at the command line. It'd be nice to have a unified single console app with lots of commands, but existing scripts could just be converted as desired.

Also, we could just say composer require drupal/console because a lot of the problems have been solved there. :-)

As for symfony/finder, it's very convenient but yah, no real attachment to it. The problem here is how to do discovery before booting the kernel (since some commands won't need the kernel). We also want to search in vendor/, or come up with some elegant discovery mechanism that includes vendor/, so that stand-alone commands can just be required using composer.

Drupal 8.1.0-beta1 → was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

@Mile23 we implement code on the discovery part to decide which commands to load based on drupal is booted or installed, also this implementation allows for custom, and contrib modules to provide commands when installed, we are about to merge a PR to allow themes to provide commands as well and we are currently working on support load commands based on arbitrary location like ~/.console or d8.dev/console

Why do we need discovery mechanism at all? Drupal core itself needs just a few console commands which can be registered statically, as usual. For other consumers (custom and contributed modules) there is Drupal Console. Furthermore Drush is also going to provide discovery mechanism using annotated command.

I propose we instantiate all core console commands explicitly in a single Symfony Console application and run them like this: bin/drupal run-tests

We already require symfony/console for core, so that we can make commands like that using the console framework. The main problem is that you then have a lot of repeat effort for things like output styling, for instance.

I already refactored run-tests #2624926: Refactor run-tests.sh for Console component. → but no one seems to care. :-)

Suppose discovery is needed cos commands should be able to use DI

I am curious to know if we have some inventory of core/scripts or respective documentaion. How many of them need to be ported to Sympfony Console commands?

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

Drupal 8.2.0-beta1 → was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

'Need' is a strong word. :-)

Updated for 8.5.x and with some coding standards stuff. Probably more is needed.

Re-rolled because of #2906637-30: [META] Drush and core compatibility is fragile →

:-)

Based on that issue, adding a command-line updater would be warranted.

There also might be some hope for composer-averse users as well, but, for instance, we should not ever add composer/composer to our list of production requirements (dev maybe).

This patch contains a cache clear command, and a run cron command, so reviewers can get a sense of how it works. Here's what it looks like when you run those commands:

$ php ./core/console drupal:cron
Running cron...
Done.
$ php ./core/console drupal:clear-cache

                                                                  
  [Symfony\Component\Console\Exception\CommandNotFoundException]  
  Command "drupal:clear-cache" is not defined.                    
  Did you mean one of these?                                      
      drupal:cache-clear                                          
      drupal:cron                                                 
                                                                  

Grant:drupal paul$ php ./core/console drupal:cache-clear
Caches cleared.
$

./core/console drupal:cron

I wonder if it is a good idea to create ./bin or ./core/bin directory for command line scripts?

we already have "core/scripts" so why not use this way?

On other hand the core's commands could use composer https://getcomposer.org/doc/articles/scripts.md#writing-custom-commands

+++ b/core/core.console.services.yml
@@ -0,0 +1,24 @@
+services:     ¶

+++ b/core/tests/Drupal/Tests/Core/Console/ApplicationTest.php
@@ -0,0 +1,23 @@
+ * ¶

+++ b/core/tests/Drupal/Tests/Core/Console/Command/RunCronTest.php
@@ -0,0 +1,85 @@
+ * ¶
...
+ * ¶

trailing whitespaces

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

I wonder if it is a good idea to create ./bin or ./core/bin directory for command line scripts?

We'd want to place console into a reasonable place, and then specify its bin status in core/composer.json. Then Composer will alias it somewhere when we use it. (That's how, for instance, phpunit ends up in vendor/bin/phpunit.)

Moved console script to core/scripts.

Eventually, we'll have to figure out how to specify bin status for core/scripts/console. We can't do that now because of the way we merge core/composer.json with the wikimedia merge plugin. The alternative is to do it in the root composer.json file, but that's not really where it should be.

This patch needs a reroll, it doesn't apply anymore. So I couldn't test this out, I did find some nits to pick though.

1. +++ b/core/console
   @@ -0,0 +1,31 @@
   + * @file
   + * Contains the Drupal console.
   

   I'm not sure about calling this drupal console, this might be really confusing.

   I think namespacing it as console is not a problem, because then we can call it "core console".

   So we can probably use Contains the core console script. here?

2. +++ b/core/core.console.services.yml
   @@ -0,0 +1,24 @@
   +services:     ¶
   

   Trailing space should be removed.

3. +++ b/core/lib/Drupal/Core/Console/Bootstrap.php
   @@ -0,0 +1,57 @@
   +  protected $classLoader;
   

   We should give this variable a docblock.

4. +++ b/core/lib/Drupal/Core/Console/Bootstrap.php
   @@ -0,0 +1,57 @@
   +  public function __construct($class_loader) {
   

   Can we typehint this? Also needs docblock.

5. +++ b/core/lib/Drupal/Core/Console/Bootstrap.php
   @@ -0,0 +1,57 @@
   +    } catch (\Exception $e) {
   

   Should be on the next line.

6. +++ b/core/lib/Drupal/Core/Console/Bootstrap.php
   @@ -0,0 +1,57 @@
   +        'Insufficient Drupal to proceed.',
   +        'This command requires a bootable Drupal installation.',
   

   I love this, insufficient Drupal. I'm not sure if it's clear enough for everyone though. Should we say that we need to have an installed Drupal?

7. +++ b/core/lib/Drupal/Core/Console/BootstrapInterface.php
   @@ -0,0 +1,38 @@
   +/**
   + * @file
   + * Contains \Drupal\Core\Console\BootstrapInterface.
   + */
   

   This should be removed.

8. +++ b/core/lib/Drupal/Core/Console/Command/ClearCache.php
   @@ -0,0 +1,37 @@
   +      // @todo: This doesn't actually clear all caches.
   +      $cache = $kernel->getContainer()->get('cache.default');
   +      $cache->deleteAll();
   

   drupal_rebuild(); does a rebuild of all caches, let's use that instead.

9. +++ b/core/lib/Drupal/Core/Console/Command/CommandBootstrapBase.php
   @@ -0,0 +1,49 @@
   + * It's very important that we don't bootstrap Drupal before the subclass asks
   + * for it.
   

   Sure, I believe ya'll, but instead of saying it's important, can we say why it's important?

10. +++ b/core/lib/Drupal/Core/Console/Command/CommandBootstrapBase.php
    @@ -0,0 +1,49 @@
    +    $this
    +      ->addOption(
    +        'environment', 'e', InputOption::VALUE_REQUIRED, 'Kernel environment.', 'prod'
    +    );
    

    This looks really weird, I don't think we use this style very often.

    Let's put this on one line instead?

11. +++ b/core/lib/Drupal/Core/Console/ServicesFinder.php
    @@ -0,0 +1,24 @@
    +class ServicesFinder extends DrupalFinder {
    

    Needs a docblock.

12. +++ b/core/lib/Drupal/Core/Discovery/DrupalFinder.php
    @@ -0,0 +1,95 @@
    +/**
    + * @file
    + * Contains \Drupal\Core\Discovery\DrupalFinder.
    + */
    

    Should be removed.

Hah. I made the changes for #91 but never uploaded the patch.

Interdiff here is to #88.

This testbot build might break because composer.lock isn't up to date. That's because of strange vagarities of the computer I'm using necessitates modifying a bunch of stuff that means a wrong content hash.

The last submitted patch, 94: 2242947_94.patch, failed testing. View results →
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Making ClearCacheTest pass after this change involves converting it to a KernelTestBase test, (because drupal_rebuild() is not actually mockable (meaning not testable)), which also involves creating a KTB test base class for the console app.

This is all work I'll gladly do if this moves forward, but the original issue was filed in 2014 and there's a year-and-a-half gap before #88. There's enough here to evaluate whether it's a thing anyone else wants.

#96, should we open up a new issue in the ideas queue? In any case, I really think that having a couple of commands available trough core would be an awesome thing. All I really need on production machines is 4 commands, so using that instead of the entire drush or drupalconsole packages would be nice.

It also makes it a lot easier for people just getting started. I am +1

+++ b/core/lib/Drupal/Core/Console/Command/ClearCache.php
@@ -0,0 +1,45 @@
+      $output->write('<error>Unable to clear caches.</error>');

We can use stderr on such occasions and return non-zero exit code.

$output->getErrorOutput()->writeln('<error>Unable to clear caches.</error>');
return 1;

This worth noting that we've already implemented a few Symfony console commands and applications. Should the console.app include them as well?
https://github.com/drupal/drupal/tree/8.5.0-rc1/core/lib/Drupal/Core/Com...

...Except that the reason we get to that code path is that there's no kernel. And if the kernel can't be generated, we already do this:

+++ b/core/lib/Drupal/Core/Console/Bootstrap.php
@@ -0,0 +1,70 @@
+    catch (\Exception $e) {
+      /** @var \Symfony\Component\Console\Helper\FormatterHelper $formatter */
+      $formatter = $command->getHelperSet()->get('formatter');
+      $error_messages = array(
+        'Insufficient Drupal to proceed.',
+        'This command must be run within an installed Drupal.',
+        $e->getMessage(),
+      );
+      $formatted_block = $formatter->formatBlock($error_messages, 'error', TRUE);
+      $output->writeln($formatted_block);
+    }
+    return FALSE;

We should implement a style object in order to pipe the message to stderr.

Soft-postpone on #2911319: Provide a single command to install & run Drupal →

Updating IS with some implementation details.

Over in #2911319: Provide a single command to install & run Drupal → we're having a debate about what to call that script. I see this issue has gone for core/scripts/console. I think it'd be great if we could agree that core/scripts/drupal is the way to go. This is inline with other CMS's for example CraftCMS (which provides a console command ootb) and makes sense as you are running commands to interact with the Drupal application. Input over on #2911319: Provide a single command to install & run Drupal → appreciated.

Better structure for the issue summary.

Definitely needs to use SymfonyStyle.

I really disagree with

1. The command name should not be called "drupal" because a lot of existing sites use Drupal console, and the Core "drupal" would conflict with the Drupal Console "drupal", both of which would install to vendor/bin.

. It makes Drupal harder to get out-of-the-box.

To quote myself from #2911319-169: Provide a single command to install & run Drupal →

@Mile23 well the aim of #2242947: Integrate Symfony Console component to natively support command line operations → is to build a generic Drupal CLI provided by core so in my mind it is that issues job to sort out the clash with drupal/console and ponder the composer bin questions. All we should do here is worry about what we are telling people to do here and unfortunately drupal-installer quick-start does not make sense. Also yes it is not nice but Drupal is this projects name and console calling their command drupal is actually the clashing thing - there already was core/scripts/drupal.sh and has been for years. There's no way around it that the obvious name for a drupal cli tool provided by core is drupal - it's what anyone would expect.

@alexpott The issue here is that if Drupal core defined a command called "drupal", then any existing site that includes Drupal Console already (e.g. any site build with drupal-composer/drupal-project) will already have a file `vendor/bin/drupal`. When one of these sites use `composer update` to pull in the new version of Drupal core that also contains a `vendor/bin/drupal`, you will not be able to predict which `drupal` Composer will put in `vendor/bin`.

@greg.1.anderson I understand that. But how about we allow console to add commands to the drupal command provided by core? And when we do that console updates to that. The still have drupal, core has drupal, drush could have drupal too.

Thanks for the update, @greg.1.anderson. :-)

Work on #2911319: Provide a single command to install & run Drupal → shows that we're comfortable with one-off scripts using console to provide UX. There are other console-based scripts in core which aren't really user-facing so much, like #2926633: Provide a script to install a Drupal testsite → which exists for tests.

That leaves us with a few decisions:

• Should extensions be able to define console commands? My answer: Yes.
• Should we, for instance, make the console a dev requirement that can be uninstalled in production? Maybe have a dev 'component' and a non-dev 'component', so that for instance the new demo installer script can always be present even if maybe-harmful commands are not.
• What testing standards should we implement for console scripts, whether they are part of an integration or stand-alone? We should have standardized ways of testing commands, so that we can use them uniformly in all scripts.

@alexpott I can't really mediate the use of the "drupal" namespace between core and console, but your idea is certainly technically possible. If you or any gatekeeper have any requirements to get this patch committed (e.g. a way for non-modules to add commands, maybe a way to do so and still participate in dependency injection, for example), then let me know, and I'll try to move this patch forward.

I might recommend, though, that the first version of a core cli released, at a minimum not use "drupal" as the command name. If this was done before Console had a chance to adjust, then everyone's CI scripts et. al. that depended on console would break. There was pretty strong consensus in the BoF that an abrupt b/c break of this sort would be undesirable.

@greg.1.anderson there is another option. We continue to use core/scripts for core scripts and build the drupal command there. And move to vendor/bin once command integration is possible and console has had time to adjust and integrate. Since Drupal is never installed in vendor at the moment and it's current scripts are not moved to vendor/bin to me this feels a good solution that allows us to move in small steps.

The BoF spent little time talking about the name 'drupal' versus other names. I would not characterize the room's tone as strong consensus that core use any name other than 'drupal'.

Note from the "CLI in Core BoF"

1. Commands in Core is a good idea. Some folks still hold reservations.
2. It is tbd how and when contrib modules should implement their commands as core extensions instead of as a Drush or Drupal Console extension. For the first release we should document this facility as "@internal" for at least one minor release.
3. Converting the existing core scripts would be a good place to start. It's a net benefit to be able to be able to get a list of all of the available cli commands available in core. Existing scripts could remain with their original name, but they would just call through to the new commands.
4. We need to be careful about how we promote it, so that folks realize that this does not replace (does not have as much functionality as) existing tools. Adding a third way to define module commands will be confusing for users in the short term.
5. The command name should not be called "drupal" because a lot of existing sites use Drupal console, and the Core "drupal" would conflict with the Drupal Console "drupal", both of which would install to vendor/bin.
6. It's unclear who would actually implement e.g. the port of existing Drupal Console / Drush commands, and whether this would gain traction.

EDITED only to fix numbering.

Yes, @moshe is correct; I mis-characterized item 5 in the summary in #107. It was not a goal of the BoF to decide what the script's name should be. There was a discussion about the problems that would arise if both core and Drupal Console installed a "drupal" script to /vendor/bin.

Talking about the name drupal versus other names was not the goal for sure @moshe weitzman since we covered several other points as mentioned by @greg.1.anderson on #107 but we all agree is not a good idea to reuse the same binary name drupal that is already used and had been used for a long time on another project drupal/console.

Making a change like this for drupal/console project is not possible. A change like this will break compatibility with current mayor release 1.x and people has been using the tool for almost 4 years now.

@alexpott related to:

Your comment on #108 How using a different name cause this:

"It makes Drupal harder to get out-of-the-box."

We can use bin/console as binary name as Symfony defined by default.

Your comment on #110

But how about we allow console to add commands to the drupal command provided by core? And when we do that console updates to that. The still have drupal, core has drupal, drush could have drupal too.

Making the “drupal” core binary able to load drupal/console commands is out of the scope for this issue as first attempt to add a CLI to Drupal Core.

If the script is in core/scripts and we don't put it into vendor/bin then we are fine too. Drupal core is not moving to vendor anytime soon and as for the name drupal as a shell command core/scripts/drupal.sh has been around for 11 years.

@alexpott The name core/scripts/drupal.sh does not cause any conflict. The issue is in the case core will decide to use the vendor/bin/drupal for the binary file name.

I don't think core should move anything into the vendor namespace until (a) we support that better for all of core code (b) the CLI runner can harness both drush and console commands with a bridge or something. At the moment core's scripts go in core/scripts. There's no need for us to change that in this issue.

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

Reroll #94 patch for 8.7.x

The last submitted patch, 74: 2242947_74.patch, failed testing. View results →

The last submitted patch, 74: 2242947_74_core_only_do_not_test.patch, failed testing. View results →

The last submitted patch, 121: 2242247_121.patch, failed testing. View results →
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

The console implementation doesn't work fine. Currently the command clear cache fail int the test.
I try to improve this but I can't use the console commands that was created.

I try to run this:

$ php ./core/console drupal:cron

(not work)

$ core/console

(not work)

How to run the console?

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

This worked for me just now:

$ git apply 2242247_121.patch 
$ git checkout -- composer.lock
$ composer update symfony/finder
$ ./core/scripts/console drupal:cache-clear

This issue is really old and doesn't have a lot of consensus, so it's probably not worth working on it very hard.

So this issue probably either needs an issue summary update or needs to be considered 'closed' and some new issues opened.

We have a symfony console CLI in core now. It lives at /core/scripts/drupal . and currently does not conflict with /vendor/bin/drupal from symfony console. If we need to solve that, we should do it in another issue.

We add two different types of console commands: Those which require a booted Drupal, and those which don't

The current CLI app supports "those which don't" - there are patches on this issue which were attempting to support 'those that do'. I believe we should open a new issue to add that bootstraping command separately.

Once we have a way to do bootstrapping commands with core natively, we can clean up the junk store of scripts we currently have: #2289405: [META] Port all shell scripts to console commands →

But, IMO, we've solved the issue title "Integrate Symfony Console component to natively support command line operations" -> that is done.

But, IMO, we've solved the issue title "Integrate Symfony Console component to natively support command line operations" -> that is done.

It all depends on what we mean by 'integrate.' The patch gives us a few different behaviors that we currently don't have:

• Booted Drupal, as you mention.
• Discovery of commands as services, even in extensions. Meaning to add stuff you just make a command class and put it in the right place.

So if those are valuable parts of integration, then we can rescope here to add those. Followups might be needed to convert existing commands to this API.

It might also be true that if we want to move forward on this, we'd need to go through the core ideas process → , which didn't exist when this issue was first filed.

Just a straight reroll for now, need to dig deeper about why to build container each time

It's a separate container. It should not use any core services.

Making the tests pass.

Refactored the cache clear command a little so it's easier to unit test. These commands are mostly proof-of-concept, so we don't need a lot of coverage at this point.

Fixed deprecated PHPUnit mocking.

Fixed CS.

The last submitted patch, 136: 2242947_136.patch, failed testing. View results →

Though the service finder looks for services inside /modules directory, it is not possible to create a console command inside a module because module namespaces are not registered in class loader.

Here is an alternative approach to register commands. The command discovery is based on class namespaces and it is similar to Symfony 3 implementation which has been removed in Symfony 4.

That patch has no tests yet just PoC.

Ignore.

Removed unused service definition.

I haven't figured out a good way to discover commands form vendor directory.

I like the discovery technique in #141. DrupalKernel::compileContainer() is already doing us the favor of collecting all of the module namespaces, plus the namespaces in Core and Components, so this seems like a natural facility for us to leverage. It is helpful that this mechanism works even without bootstrapping Drupal, so that commands that don't need a bootstrap can be executed quickly.

I think it is also important to allow commands to be constructed via the service container. Both Drush and Drupal Console currently do this; it is very useful for commands that already require a Drupal bootstrap to be able to utilize Dependency Injection for their commands.

It's also great that it is possible in #141 to make DrupalCommandInterface commands that do not require a bootstrap. It is possible to have the best of both mechanisms if the command dispatcher delays bootstrapping Drupal until necessary. Drush does this by extending the Symfony Console Application to bootstrap further in Application::find(), so that more commands can be found if needed. Of course, the downside is that "command not found" and "help" both need to bootstrap Drupal.

Some other comments about the issue summary:

This CLI integration should be viewed as a separate application which ships with Drupal. This might include going so far as to turn it into a subtree split similar to Drupal components.

Providing the CLI integration via a subtree split is not a good thing to do. This would create a separate global CLI application that would ostensibly be able to target multiple different Drupal sites. Both Drush and Drupal Console have shown that this is a Hard Problem. It is much better to consider the CLI integration to be a separate front controller on the SAME Drupal application / site. Bundling the CLI front controller with Drupal is the primary benefit to having the CLI in core; we don't really need Yet Another Global CLI to bolt onto the side of Drupal. We already have enough of those, and they work well.

Figure out the name. drupal/console project is already using the name 'drupal' as its bin file. Arm wrestle over it?

As pointed out in #131, core/scripts/drupal is the Drupal core CLI, and vendor/bin/drupal is Drupal Console (if it has been added to the site). There is no need for core/scripts/drupal to go in the user's global $PATH. Users might wish to put relative paths core/scripts and vendor/bin in their $PATH, in which case they could use drupal to run either Drupal Console or the core Drupal CLI per their preference. Most users will probably directly address the scripts (e.g. `core/scripts/drupal commandname`) or use a launcher. I therefore do not think this is a technical issue that needs to be solved in this issue.

I'd be happy to update the issue summary if there is general agreement on those points (or someone else can do it).

Oh yeah I forgot to address #142. I don't think we really need to be able to run commands out of vendor at this point. It should be sufficient to provide a way to add commands to modules. If we did want Yet Another Way to define commands in Drupal, though, Robo provides an extension mechanism based on the psr4 autoloader and well-known namespaces that could be referenced for inspiration.

I think that #141 is the better option for Drupal core, though.

+1 on #143, #144, and the good work up to and including #141. I had concerns about a bootstrapped DrupalKernel as in #141. We might even switch to use the installer kernel just to do autoloading, but it might not matter.

We might even somehow factor the extension namespace loader functionality out into its own domain, so we can just do that without a kernel. That might be too radical a change, though.

I like the idea of discovery in vendor better than I like the idea of discovery in modules. :-) But I'm mostly ambivalent about it. The main goal is to have functionality in core, that tests along with it, that doesn't fall out of sync with dependencies and always works.

I was looking at Robo to find out how you guys did the extension system. I think the Robo model for that might be the best for having commands in vendor, so we can use a \MyNamespace\Drupal\Console\Command\ThisIsTheCommandClass type naming convention. But again, somewhat ambivalent as long as we can come up with something that core maintainers might like and/or enjoy.

Allowing module commands to be able to participate in dependency injection is an important feature. If we can somehow do that with vendor-discovered command classes, then I'm all in favor of that route.

Dependencies can be injected via create factory method. Just like in plugins, controllers and forms.

I would like to have command discovery in vendor though it won't be a trivial task.
Firstly it requires a reliable way to figure out the location of vendor directory. The patch in #136 assumes that vendor directory is inside Drupal root, but it's kind of outdated because nowadays Composer based Drupal installations are ubiquitous.
Secondly, the discovery command with refection is not applicable to vendor packages because their namespaces are unknown. It's tricky in PHP to figure out class by knowing the filename. We probably can introduce some predefined namespace as was mentioned above. Another approach could be declaring command classes explicitly in package composer.json file under extra section.

Here is a relevant issue in Drush.
https://github.com/drush-ops/drush/issues/3050

Also each command should indicate weather or not it needs fully bootstrapped Drupal. I think we can add DrupalAwareInterface for this purpose.

I would like to have command discovery in vendor though it won't be a trivial task.

We don't need to do file discovery. If we have the class loader, we can ask it for all the namespaces and then figure out if any of them contain a [YourNamespace]\Drupal\Console\CommandFactory class that implements \Drupal\Core\Console\CommandFactoryInterface.

Then we instantiate the factory and it gives us all the commands.

And since we're using a booted DrupalKernel and using its container, we can then move on to just tagging services as console_command or whatever, and discovering them that way, and that's how we discover commands in modules.

So that's two cases: vendor we look for classes with magic names, and modules and core define a service with a special tag. That's not too complicated, is it?

If we have the class loader, we can ask it for all the namespaces

I could not find an appropriate method for this.
https://github.com/composer/composer/blob/master/src/Composer/Autoload/C...

Interesting, if define a command as service like this, class resolver can instantiate it from container with proper dependency injection.

services:
  Drupal\foo\Command\FooCommand:
    arguments: ['@entity_type.manager']

Yeah, okay, I could be +1 on using #152 with #147 -- forget about services and just pass the container into a static factory (create) method and let command classes pull whatever dependencies they need.

I don't think we need a marker interfaces for bootstrapping; I think it would be better to just have a bootstrapping trait (or even class) that commands could use to bootstrap if they need to. Previous patches on this issue had bootstrapping in a base class, but I think it would be better to keep the bootstrap utility out of the inheritance tree. Minor point, just code style.

[duplicate]

I think it would be better to just have a bootstrapping trait (or even class) that commands could use to bootstrap if they need to.

I suppose about 90% of custom and contrib commands will need it. Forcing each of them to boostrap Drupal manually will bring a good amount of duplicated code. And besides, in order to register commands from installed modules we have to bootstrap Drupal anyway.

Another point is that instantiation of commands that have dependency injection via constructor is not safe when Drupal is not fully bootstraped.

I could not find an appropriate method for this.

The RelativeNamespaceDiscovery::getClasses() from Robo works perfectly though. However, it brings back the dependency on symfony/filesystem component.

Drush only instantiates commands from enabled modules, for safety. I didn't try to instantiate commands without a bootstrap, but I agree with #155 that it would be safer to not do this. It might therefore be better to stick with established conventions and always bootstrap for module commands. Then, most commands won't need to worry about bootstrapping.

The attached patch takes a different approach. Commands needs to be registered explicitly in composer.json files similar to Composer scripts.

"extra": {
    "commands": [
        "Drupal\\foo\\Command\\BarCommand"
    ]
}

Here is a brief summary of the patch:

1. The command discovery works for core, enabled modules and themes, vendor packages.
2. Commands that needs a fully bootstrapped Drupal installation must implement empty DrupalAwareInterface.
3. For "Drupal aware" commands dependencies can be injected via create factory method. We may consider creating a separate class resolver to support DI for all commands.
4. For modules it is possible to define commands as a services (see note #151).
5. The implementation is less magic because it does not rely on reflection, well-known name spaces and file naming conventions.
6. The patch does not bring new dependencies to core.
7. It's possible to register global (Side Wide) commands that do not belong to a module in main composer.json file.

It hasn't been discussed yet. Do CLI commands need to be translatable? I think Drush 9 does not support it. On the other side Drupal Console team had worked hard to make commands multilingual.

The last submitted patch, 157: composer_commands-2242947-157.patch, failed testing. View results →
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Re #158 Drupal's ability to work in multiple languages is one of its strongest features. I would say that we probably definitely want to be able to translate commands.

OK, so how do we implement multilingual for a site that isn't installed, so that you can install it from the command line?

Maybe we need to go back to the installer kernel idea for command discovery. This gives us services and allows a multilingual install process without an existing site.

Then commands that need an installed Drupal to work can instantiate their own DrupalKernel (via a base class which we provide).

-1 on adding to composer.json extra sections. That's for Composer scripts, not our use case. It would also require too much setup for a given command to work out of the box.

Here's a patch that uses DrupalKernel to initialize the container enough to discover services, and then assemble the console app with the tagged commands. Moving to using translation will be enabled by getting @string_translation through the container.

It can't discover services for modules which aren't enabled, because why would it? :-) There's a POC of this for the clear cache command, which I put in the system module. If you perform an installation, the system module will be available, and its command is discovered.

It does not discover any commands in vendor, because there are only so many hours in a day. The plan might well be to add another bit to the ConsoleServiceProvider which does whatever scan is necessary.

I spent a little time poking through drupal/console looking at their translation system. It's not using Drupal core, but it's good for them. I'm still trying to figure out where the actual string swaps occur, because that's where ours should, as well.

Interdiff from #136.

Edit: Woops. Left over service definitions from the old patch. Will need to change that.

It would also require too much setup for a given command to work out of the box.

Why? It is as simple as copying class name to composer.json file. Same needs to be done with service based commands. A developer has to configure new command in console.services.yml file.

In #162 we're not using console.services.yml. It's the kernel's container.

I could have sworn I changed that file name....

The last submitted patch, 165: 2242947_165.patch, failed testing. View results →

This demonstrates locating services by using the classloader to find service providers with magic names.

If you apply this patch locally, be sure and say composer dumpautoload before you do php ./core/scripts/console list. You'll see drupal:cron in the list, which is discovered through a magic-named service provider factory type thing. :-)

The tests will still fail because I didn't fix them here.

There are plenty of coding standards errors, but this is PoC.

The downside here is that in order to provide a command from a dependency that isn't a module, you have to know how to make a Symfony DIC service definition in code. It's not terribly hard, but it's less convenient than YAML. We could provide a boilerplate service provider that just reads in an arbitrary service definition YAML file from within the package in vendor. That would make it easier for DX.

The last submitted patch, 167: 2242947_167.patch, failed testing. View results →
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

I am not fond of #157. Searching for composer.json files and aggregating information from them reminds me of the old Composer Manager. The right way to get info from the `extra` section is via the Composer APIs, but those are only available to plugins, and it wouldn't make sense for Drupal core commands to be Composer plugins.

I like the direction that #167 is going. I have a couple minor comments, but perhaps the items below are only the way they are because this is a PoC.

1. +++ b/core/lib/Drupal/Core/DrupalKernel.php
   @@ -617,6 +618,10 @@ public function discoverServiceProviders() {
   +    if ($this->environment === 'console') {
   

   It would be better if the ConsoleServiceProvider were somehow injected by the Console front controller rather than having a conditional here.

2. +++ b/core/lib/Drupal/DrupalConsole/CommandServiceDefinition.php
   @@ -0,0 +1,26 @@
   +  public function getDefinitions() {
   

   This isn't really a magic name; this is an explicit declaration of a command definition. I'm not sure how a module would extend this. I think it would be possible to replace this function with something that searched for well-known (magic) names though.

The right way to get info from the `extra` section is via the Composer APIs

Correct, but it's not forbidden to read that information directly. Drush does it and Drupal core possibly will do it at some point.
✨ Replace .info.yml with composer.json for extensions Postponed

I do not dispute that it is possible to use composer.json for such purposes; there are use cases where it is reasonable to put non-Composer metadata in composer.json. However, it's not the best idea. I am skeptical about ✨ Replace .info.yml with composer.json for extensions Postponed , and we probably would have put the Drush metadata in the info.yml file instead of the composer.json file but for the fact that composer.json has extra and info.yml files do not.

The larger issue with #157 is not that it reads info from composer.json as much as that it has to search the filesystem for composer.json files and then aggregate information from them. Discovery should happen using information already available to use. There's an example above that does it, but I'll have to look at it later.

As another example, Robo's plugin mechanism uses $this->classLoader->getPrefixesPsr4() as $baseNamespace => $directories to find the set of namespaces to search, and then uses a filesystem search over just a small subset of the filesystem. This is much better then searching the entire modules directory. It could be improved even further by making a single well-known class in each namespace that declared and returned the list of classes with commands. I wish it had been implemented this way (maybe for Robo 2.x?). I similarly regret the command discovery class of the annotated command library, and will probably replace that with something better in the future.

Re: #169:

1: We can't inject the service provider because the kernel doesn't give us any opportunity to.

2: The function isn't a magic name. The magic name is the name of the class we discover. I guess I should call it a naming convention.

We can't inject the service provider because the kernel doesn't give us any opportunity to.

But we are core. Can we add a setter to the Kernel?

The magic name is the name of the class we discover.

How would a module add a command with this mechanism? It looks like all of the command names (just one ATM) are listed in code. This is fine for core commands, but we need discovery for modules. Maybe I'm just not seeing it.

Why should the console main application be registered as a service ? Symfony does not do this, and just:

$kernel = new Kernel($env, $debug);
$application = new Application($kernel);
$application->run($input);

Of course, it needs to be adapted to Drupal, but it seems much saner.

Why wouldn't Drupal have two different binaries, one "console" that work on a working site, and another such as "install" or "maintainance" that does everything that will need to run over a non bootstrapped site, spawning a generic application instance without the kernel (you can use the console component without any kernel). Then you just separate both types of commands, in the maintainance script you just add them to the application manually calling successively add():

use Drupal\...bla\Unbootstrapped1Command;
use Drupal\...bla\Unbootstrapped2Command;
use Symfony\Component\Console\Application;
use Symfony\Component\Console\Input\ArgvInput;

require_once dirname(__DIR__).'/vendor/autoload.php';

$application = new Application("Drupal core - maintainance");
$application->add(new Unbootstrapped1Command());
$application->add(new Unbootstrapped2Command());
$application->run(new ArgvInput());

And in the other script, you just bootstrap the DrupalKernel, inject it into a new Application instance, and run it.

This way it would make your life much simpler, ask people to tag using console.command their commands and register it into a Symfony\Component\Console\CommandLoader\ContainerCommandLoader instance, so they end up into the container and your application will discover them much transparently and you're good to go.

Can we add a setter to the Kernel?

We want to perform discovery and container manipulation during the container compile time. DrupalKernel has that roped off, and it'd be pretty complex to change that. Also, as you say: We're core, so we can add a conditional. :-)

How would a module add a command with this mechanism?

I guess it might not be obvious from the patch.

In #167, if you want to add a command to your module, you add it as a service to your modulename.services.yml file and tag it with console.command. If the module is enabled, the command is discovered. This is just like any other service you'd get from a module.

If you want to make a composer-based package instead, you use a class named YourNamespace\DrupalConsole\CommandServiceDefinition that implements DrupalConsoleServiceDefinitionProvider. That ends up looking like this:

  public function getDefinitions() {
    return [
      // Service name.
      'console.run_cron.command' =>
        // Service definition based on class name.
        new Definition(RunCron::class, [
          // References to other services, equivalent to arguments: in YAML.
          new Reference('console.bootstrapper'),
        ]),
    ];
  }

As I mentioned in #167, we could provide some boilerplate that parses a YAML file and then satisfies DrupalConsoleServiceDefinitionProvider, and then people could just use YAML.

The larger issue with #157 is not that it reads info from composer.json as much as that it has to search the filesystem for composer.json files and then aggregate information from them.

Both glob and file_exists are quite fast in PHP and #157 does not use recursive file search. I've just made a package from it to ease testing and benchmarking. On brand new Drupal 8 installation getComposerFiles() takes only 12ms. On a real site with lots of modules and vendor packages the time rises to 15ms which is perfectly acceptable for CLI application.

Discovery should happen using information already available to use.

The problem here is that we don't have enough information. Class loader collects prefixes not namespaces. Robo extension discovery relies on symfony/filesystem to identify namespaces. And without it even class_exists() may force class loader to call file_exist().

Why wouldn't Drupal have two different binaries, one "console" that work on a working site, and another such as "install" or "maintainance"

Technically we can make two different applications that share the same binary.

Re: #170#171 The general consensus on ✨ Replace .info.yml with composer.json for extensions Postponed is that we are not planning on it being an either/or scenario. The plan is to only deprecate the core key, and the dependencies/test_dependencies keys, and leave .info.yml files, the bulk of that work is actually being done in #3005229: Provide optional support for using composer.json for dependency metadata → .

This is because we are not interested in overloading composer.json to be a metadata storage facility, and instead only want to rely on it for dependency resolution purposes. .yml files are vastly superior to .json files for configuration.

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule → and the Allowed changes during the Drupal 8 and 9 release cycles → .

Class loader collects prefixes not namespaces.

It collects whole class names if they're part of a classmap. Otherwise, yes, it's prefixes. But our naming convention would use the prefix to discover the class. That's what #167 is doing.

So a prefix like Mile23\Something\ gets reduced to Mile23\ in order to figure out if Mile23\DrupalConsole\ConsoleServiceProvider exists. If the service provider doesn't have an appropriate prefix in PSR-0/4 or classmap, then it can't be autoloaded anyway.

We now have an idea process starting up for CLI in core type stuff: #2242947: Integrate Symfony Console component to natively support command line operations →

Woops. Wrong link.

#3089277: Provide core CLI commands for the most common features of drush →

This is a re-roll of #167 against Drupal 9.0.x.

This is presented so it's easier to evaluate what's going on here. Real work is happening in #3089277: Provide core CLI commands for the most common features of drush → and not here. Marking this issue postponed.

This does not address any of the comments since #167 (and maybe not even some before that).

Targeting 9.0.x because it's super duper highly very improbably unlikely that any of this will end up in core before then.

That's the old patch.

Here is the new one.

Here are some of my observations:

1. +++ b/core/lib/Drupal/Core/Console/Bootstrap.php
   @@ -0,0 +1,70 @@
   +      $kernel = DrupalKernel::createFromRequest($request, $this->classLoader, $env);
   +      $kernel->boot();
   ...
   +      // @todo: Change this when we're not using includes any more.
   +      require_once dirname(dirname(dirname(dirname(__DIR__)))) . '/includes/common.inc';
   

   I think we should call $kernel->preHandle($request); here to load all the modules and includes. Then we can remove this require once as well.

2. +++ b/core/lib/Drupal/Core/Console/Bootstrap.php
   @@ -0,0 +1,70 @@
   +      $container->set('request', $request);
   

   I don't think this is need. We don't use ->get('request') anymore and I was not able to find any occurrence in core as well.

3. +++ b/core/lib/Drupal/Core/Console/Bootstrap.php
   @@ -0,0 +1,70 @@
   +      $container->get('request_stack')->push($request);
   

   We should also set $container->get('router.request_context')->fromRequest($this->request); here.

4. +++ b/core/modules/system/src/Command/ClearCache.php
   @@ -0,0 +1,53 @@
   +      ->setAliases(['cc'])
   

   We should set 'cr' as well.

5. +++ b/core/modules/system/src/Command/ClearCache.php
   @@ -0,0 +1,53 @@
   +    drupal_rebuild($container->get('class_loader'), $container->get('request'));
   

   Why can't we get this request from request stack?

+++ b/core/lib/Drupal/Core/Console/Command/CommandBootstrapBase.php
@@ -0,0 +1,45 @@
+abstract class CommandBootstrapBase extends Command {

+++ b/core/lib/Drupal/Core/Console/Command/RunCron.php
@@ -0,0 +1,38 @@
+class RunCron extends CommandBootstrapBase {
...
+    $kernel = $this->bootstrap->bootstrap($this, $input, $output);

+++ b/core/modules/system/src/Command/ClearCache.php
@@ -0,0 +1,53 @@
+    $kernel = $this->bootstrap->bootstrap($this, $input, $output);

When do we need the concept of a bootstrapped Drupal command vs. not bootstrapped? Shouldn't it always just be bootstrapped?

+++ b/core/lib/Drupal/Core/Console/Bootstrap.php
@@ -0,0 +1,70 @@
+      // Create a meaningful request object. We shouldn't really need this but
+      // Drupal complains if it's not present.
+      $request = Request::createFromGlobals();
...
+      $kernel = DrupalKernel::createFromRequest($request, $this->classLoader, $env);
+      $kernel->boot();

This part is confusing. It looks to me like the kernel is too tightly coupled to the Http layer. Perhaps we should try fix that, but at the very least we should elaborate on why we need to do this.

#188: It would not be possible to write commands like "site install" if a full bootstrap was always necessary. It is also much faster to omit the bootstrap for commands that do not need to call the full Drupal API suite. That said, I do not know how many of the commands being considered for inclusion in core will run with a partial bootstrap.

#189

But aren't we already kinda doing that in the console script?

+++ b/core/lib/Drupal/Core/Console/Bootstrap.php
@@ -0,0 +1,70 @@
+      $kernel = DrupalKernel::createFromRequest($request, $this->classLoader, $env);
+      $kernel->boot();

+++ b/core/scripts/console
@@ -0,0 +1,26 @@
+$kernel = DrupalKernel::createFromRequest(Request::createFromGlobals(), $class_loader, 'console');
+$kernel->boot();

The existing "drupal" script does not bootstrap the kernel. The second example in #190 looks incorrect to me, but I did not try to install Drupal using this patch. Maybe someone else can explain why the wrapper script is like this.

This would be a minor-only addition. Since 8.9.x and 9.0.x are now in beta, I'm moving this to 9.1.x. Thanks!

Reroll for 9.1.x.

#3089277: Provide core CLI commands for the most common features of drush → suggests this issue should become a meta.

I'd suggest that a meta be a brand new thing where we can track the actual plan. This issue is more of a proof-of-concept.

Will set back to 'postponed' after the tests finish.

1. +++ b/core/composer.json
   @@ -21,6 +21,7 @@
   +        "symfony/finder": "~4.4",
   

   It has own issue #1451320: Evaluate Symfony2's Finder Component to simplify file handling →

2. +++ b/core/lib/Drupal/Core/Console/ConsoleServiceProvider.php
   @@ -0,0 +1,89 @@
   +      // @todo Determine a better magic naming scheme.
   +      $class = $exploded[0] . '\\DrupalConsole\\CommandServiceDefinition';
   +      // @todo Check that our class implements the interface.
   +      if (class_exists($class)) {
   +        $command_service_definiton = new $class;
   +        $definitions = $command_service_definiton->getDefinitions();
   
   +++ b/core/lib/Drupal/DrupalConsole/CommandServiceDefinition.php
   @@ -0,0 +1,26 @@
   + * Service definition provider with a magic name.
   + *
   + * This class is PoC that you can register a command this way.
   + */
   +class CommandServiceDefinition implements DrupalConsoleServiceDefinitionProvider {
   

   Nice trick, but this todos needs to be resolved

3. +++ b/core/lib/Drupal/Core/Console/DrupalConsoleServiceDefinitionProvider.php
   @@ -0,0 +1,20 @@
   + * To change this license header, choose License Headers in Project Properties.
   + * To change this template file, choose Tools | Templates
   + * and open the template in the editor.
   + */
   +
   +namespace Drupal\Core\Console;
   +
   +/**
   + * Description of DrupalConsoleServiceDefinitionProvider
   + *
   + * @author paulmitchum
   

   Looks like auto generated and needs clean-up

4. +++ b/sites/default/default.services.yml
   @@ -118,8 +118,7 @@ parameters:
   -  # get X-Drupal-Cache-Tags, X-Drupal-Cache-Contexts and X-Drupal-Cache-Max-Age
   -  # headers.
   +  # get X-Drupal-Cache-Tags and X-Drupal-Cache-Contexts headers.
   
   @@ -127,13 +126,15 @@ parameters:
   -  factory.keyvalue: {}
   +  factory.keyvalue:
   +    {}
   ...
   -  factory.keyvalue.expirable: {}
   +  factory.keyvalue.expirable:
   +    {}
   
   +++ b/sites/default/default.settings.php
   @@ -105,15 +105,13 @@
   - * Drupal core implements drivers for mysql, pgsql, and sqlite. Other drivers
   ...
   + * Transaction support is enabled by default for all drivers that support it,
   
   @@ -226,20 +224,6 @@
   - * Sample Database configuration format for a driver in a contributed module:
   - * @code
   - *   $databases['default']['default'] = [
   

   This changes looks out of scope, probably re-roll caused it

Drupal 9.1.0-alpha1 → will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule → and the Allowed changes during the Drupal 9 release cycle → .

The last submitted patch, 193: 2242947_193.patch, failed testing. View results →
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Hi Sorry for a dumb question , I can't find a drupal/console roadmap. Does drupal/console have it as a goal to be able to run commands like symfony/console? For example https://github.com/symfony/demo/blob/d2b90fbda5346022c6e92393022ef649987...

and

https://symfony.com/doc/4.2/templating/syntax.html `php bin/console lint:twig templates/article/recent_list.html.twig` ?

I'm currently having an issue with asm89/twig-lint and Drupal9. Drupal 9 has a later version of symfony/console than asm89/twig-lint. I'm just looking for a more up to date twig lint replacement.

@dshumaker Drupal core does not yet provide a generic endpoint for running console commands (i.e. bin/console). Most people are using Drush for that. As of linting Twig templates asm89/twig-lint seems abandoned. However, you should be able to install it globally. Also check out https://github.com/friendsoftwig/twigcs.

Drupal 9.4.0-alpha1 → was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule → and the Allowed changes during the Drupal core release cycle → .

Drupal 9.3.0-rc1 → was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule → and the Allowed changes during the Drupal core release cycle → .

Drupal 9.2.0-alpha1 → will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule → and the Allowed changes during the Drupal core release cycle → .

There's some confusion because there is a project called drupal/console, which is not the 'official' Drupal console.

There *is* a console command that ships with core, at /core/scripts/drupal, which is very likely where any work in this issue will end up.

Drupal Console is minimally maintained as just confirmed here by the only maintainer that's left. I think it would be a good idea to merge the Drush & Drupal Console efforts and consider using Drush as the default Drupal CLI.

#203: I don't think that Drupal core needs the full Drush for its console functionality.

We can and should discuss though what components from the consolidation namespace can be leveraged by core. I bet Moshe has an idea which make sense.

I'm not aware of anyone proposing adding Drush to Drupal core. Instead of this issue, I support ✨ Add drush/drush to Drupal Composer project templates Active .

OK, to recap:

1) There's Drush. :-)

2) There's Drupal Console, which is a third-party console app which is still minimally supported.

3) There is this issue, which really is very, very old. It was prompted by having Drush not work with a specific Drupal vs. PHP version (IIRC), so the idea was to have some of that stuff live in core, where it could be maintained alongside core itself, instead of as a third party. This issue is also from a time before the Ideas project, and should probably be marked as duplicate of....

4) #3089277: Provide core CLI commands for the most common features of drush → is an Ideas issue which wants to minimally replace some Drush behaviors with native Drupal console-based commands, without getting bogged down in implementation details as this issue does. However, it did get bogged down in the same kind of inertia that this issue did.

And now we apparently also have:

5) ✨ Add drush/drush to Drupal Composer project templates Active which isn't such a bad idea, but again suffers from a potential lag time where Drush might not be compatible with core or its dependencies for some reason. Our community's release schedule and process has improved drastically since 2014, so I doubt it's nearly as much a concern today.

Marking this issue as postponed, because if someone reeeeeeeaaaaallly needs me to re-roll #193 I will. However, effort and conversation should happen on #3089277: Provide core CLI commands for the most common features of drush → and/or it's children, which are the spiritual successors of this issue.

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened → , as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule → and the Allowed changes during the Drupal core release cycle → .

Drupal 9.5.0-beta2 → and Drupal 10.0.0-beta2 → were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule → and the Allowed changes during the Drupal core release cycle → .

Great writeup! This totally makes sense.