Contrib.social

Add a vendor:// stream wrapper

Open on Drupal.org →
Created on 26 September 2018, about 6 years ago
Updated 11 January 2024, 9 months ago

Blocked by: ✨ Implement core management of 3rd-party FE libraries Active

Drupal 8 integration with composer has been excellent for managing 3rd party PHP libraries with Drupal, as the libraries can be called using PSR-4 namespaces to load the libraries, regardless of the location of the vendor folder. Being able to call these files regardless of the location of the vendor folder is important, as the vendor library folder location is dynamic. Many installations have the vendor folder installed in the webroot, however it is a best practice to place the vendor folder outside the webroot, a practice followed by the Drupal Composer template,

The problem comes with CSS and JS libraries. These can be included using composer, but when the vendor file is outside the webroot, any CSS and JS are unusable, as they are not web-accessible and therefore cannot be included in *.libraries.yml files.

For contributed module developers, this has led to two separate methods of managing 3rd party libraries with Drupal 8. PHP libraries can and are managed with Composer, and module developers can use composer.json files to manage their dependencies. However, modules that use 3rd party CSS/JS libraries have to either use D7 methods of creating a libraries folder, and have users manage the dependencies on their own to place the library in a web-accessible location, or use Composer to manage the dependency, then have users copy the CSS and JS files to a web accessible location. An example of this is the jQuery Colorpicker → module, which as of the time of writing of this post requires users to manage the library on their own, downloading it to a /libraries folder, linking to the CSS and JS files using:

library:
  version: 1.0.1
  css:
    theme:
      /libraries/jquery_colorpicker/css/colorpicker.css: {}
  js:
    /libraries/jquery_colorpicker/js/colorpicker.js: {}
  dependencies:
    - core/jquery

As you can see, the CSS and JS files are linked to from the /libraries folder. In order to ensure system integrity, hook_requirements() has been implemented, informing users of the requirement of downloading the library and providing instructions on how to do so. This is not ideal. The ideal would be to Include a dependency on the library through Composer, managing the 3rd party library in the same manner as PHP libraries are managed.

The solution to the issue of managing 3rd party CSS and JS libraries with Composer is to use the Vendor Stream Wrapper module → . This module is essentially a clone of the private stream wrapper functionality of core, but working with the vendor directory rather than the private files directory. It sets up a vendor:// stream wrapper, and adds parsing to *.libraries.yml files, so that the above jquery_colorpicker.libraries.yml file could be converted to this:

library:
  version: 1.0.1
  css:
    theme:
      vendor://jaypan/jquery_colorpicker/css/colorpicker.css: {}
  js:
    vendor://jaypan/jquery_colorpicker/js/colorpicker.js: {}
  dependencies:
    - core/jquery

The module looks for the vendor folder first in the webroot, then one folder above the webroot. If the vendor folder is in any other location, the location can be set in settings.php.

I think this module should be included into core, to allow for Drupal modules to manage CSS and JS libraries using Composer, unifying 3rd party library management into a single API, rather than having to use D8 methodology for PHP libraries, and D7 methodology for managing CSS and JS libraries. Currently it is a bug that CSS and JS files in a vendor folder that exists outside the webroot are unable to be linked to, and it becomes a security risk if one of the files that is part of the 3rd party library is found to have a security vulnerability. This security vulnerability would not exist for users who have placed their library in a vendor folder that is outside the webroot.

✨ Feature request
Status

Postponed: needs info

Version

11.0 🔥

Component
File system  →

Last updated 1 day ago

Created by

🇨🇦Canada Jaypan

Live updates comments and jobs are added and updated live.
  • Needs issue summary update

    Issue summaries save everyone time if they are kept up-to-date. See Update issue summary task instructions.

  • Security improvements

    It makes Drupal less vulnerable to abuse or misuse. Note, this is the preferred tag, though the Security tag has a large body of issues tagged to it. Do NOT publicly disclose security vulnerabilities; contact the security team instead. Anyone (whether security team or not) can apply this tag to security improvements that do not directly present a vulnerability e.g. hardening an API to add filtering to reduce a common mistake in contributed modules.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment over 1 year ago →
    🇺🇸United States smustgrave

    Brought this up in #contribute

    @mondrake brought up there is a contrib for this https://www.drupal.org/project/vendor_stream_wrapper →
    @catch mentioned - covered by https://www.drupal.org/docs/develop/using-composer/manage-dependencies#i... → (see libraries mentioned in there).

    If still a valid feature request please reopen updating issue summary for D10

    Thanks!

  • Status changed to Active over 1 year ago
  • Comment over 1 year ago →
    🇨🇦Canada Jaypan

    @mondrake brought up there is a contrib for this https://www.drupal.org/project/vendor_stream_wrapper →
    @catch mentioned - covered by https://www.drupal.org/docs/develop/using-composer/manage-dependencies#i → ... (see libraries mentioned in there).

    Installer directories aren't and don't do the same thing as Vendor Stream Wrapper.

  • Comment over 1 year ago →
    🇬🇧United Kingdom catch

    With composer installers you can install css and js libraries inside the webroot, and PHP libraries outside the webroot, per https://www.drupal.org/docs/develop/using-composer/manage-dependencies#i... →

  • Comment over 1 year ago →
    🇬🇧United Kingdom catch
  • Comment over 1 year ago →
    🇨🇦Canada Jaypan

    With composer installers you can install css and js libraries inside the webroot, and PHP libraries outside the webroot, per https://www.drupal.org/docs/develop/using-composer/manage-dependencies#i → ...

    I believe you are incorrect, as this would require that the developer give the library some type key that identifies it as being distinct from a PHP library. As we don't have control over whether these libraries implement the type key, installers paths can NOT be used in the same way the vendor stream wrapper is used.
    That said, I'm open to being wrong. If you feel that this can be done with composer installers, how would you sort this library into the webroot, instead of the vendor folder: https://github.com/jaypan/jquery_colorpicker
    I don't believe there is a way to have that library installed into the webroot in composer.json, without having the library developer change the library.

  • Comment 9 months ago →
    🇩🇪Germany geek-merlin Freiburg, Germany

    @catch:
    > With composer installers you can install css and js libraries inside the webroot, and PHP libraries outside the webroo
    Whether #54 is a striking point for your #52, can be debated.

    As the IS states, having third-party libraries outside of the webroot is a big security hardening (eg. examples-contained-in-libraries are a well-known attack vector).

  • Comment 9 months ago →
    🇬🇧United Kingdom catch
  • Comment 9 months ago →
    🇨🇦Canada Jaypan

    @jaypan there examples on the page I linked, see https://www.drupal.org/docs/develop/using-composer/manage-dependencies#i → ...

    I don't believe those examples address this issue:

    this would require that the developer give the library some type key that identifies it as being distinct from a PHP library. As we don't have control over whether these libraries implement the type key, installers paths can NOT be used in the same way the vendor stream wrapper is used.

    Setting the directory to which a library should be installed, requires that the library itself have a specific value set for the project type that identifies it as needing to be put inside the web directory. As libraries are usually 3rd party, with site developers not having control over them, I do not see that the examples that you have linked to are able to be used to put composer-managed CSS/JS libraries into the webroot.

    Again, I'm open to being proven wrong, but the examples you have linked to, using installer-paths, do not seem to be able to cover the requirements.

  • Comment 9 months ago →
    🇬🇧United Kingdom catch

    ads via Carbon
    IONOS, Big web tech for your business
    ADS VIA CARBON
    Advertising sustains the DA. Ads are hidden for members. Join today
    On this page
    Download Drupal core using Composer
    Create a project
    Installing with composer via docker
    Install Drupal using the standard web interface
    Install Drupal using the command line
    To do a modified install
    Download contributed modules, themes and their dependencies using Composer
    Download contributed modules and themes using Composer
    Specifying a version
    Using dev versions
    About semantic versioning
    Using Composer search
    Using Composer browse
    Define the directories to which modules, themes, libraries, etc. should be downloaded
    Downloading third-party libraries using Composer
    Managing existing sites using Composer
    Download contributed modules and themes using Composer
    Updating Drupal sites using Composer
    Patching projects using Composer
    Patching Drupal core and modules
    Patching without updating Drupal core or modules
    Requiring a specific commit of a module
    Using Composer without drupal/recommended-project (advanced users)
    Moving from drupal/core-recommended to drupal/core
    Moving back to drupal/core-recommended from drupal/core
    Using Composer
    Using Composer in a Drupal project
    Composer in relation to Drush Make
    Using Composer to Install Drupal and Manage Dependencies
    Add a composer.json file
    Starting a Site Using Drupal Composer Project Templates
    Managing dependencies for a contributed project
    Managing dependencies for a custom project
    Preparing your site for Composer 2
    Troubleshooting Composer
    Using Drupal's Composer Scaffold
    Using Drupal's Lenient Composer Endpoint
    Using packages.drupal.org
    Using Composer to Install Drupal and Manage Dependencies
    Primary tabsToggle
    Edit
    Discuss
    Outline
    View history
    Last updated on 6 January 2024
    Composer can be used to manage Drupal and all dependencies (modules, themes, libraries). Make sure you have composer installed on your local machine before executing any Composer commands.

    For information on managing npm (JavaScript) development dependencies for Drupal, refer to the Frontend developer tools for Drupal core.

    Download Drupal core using Composer
    As of the release of Drupal 8.8.0 - the recommend composer template changed from drupal-composer/drupal-project to the officially supported: drupal/recommended-project

    For Drupal 10, use the composer template at drupal/recommended-project. This template ensures Drupal Core dependencies are the exact same version as the official Drupal release. Other approaches can be found below.

    For Drupal 7, use drupal-composer/drupal-project.

    Create a project
    For Drupal 10:

    composer create-project drupal/recommended-project my_site_name
    For Drupal 7:

    composer create-project drupal-composer/drupal-project:7.x-dev -n my_site_name
    This will create a project in the my_site_name directory and automatically executes composer install to download the latest stable version of Drupal and all its dependencies.

    The my_site_name directory will contain files that should be outside of your web root and not accessible by the web server. The web root will be the my_site_name/web directory, and typically the full path to that directory will be used as DocumentRoot (if you are using the Apache webserver).

    Note that the above command will download the current official release of Drupal. If you want a different version, add the version number to the command after a colon. For example, to download version 9.3.12:

    composer create-project drupal/recommended-project:9.3.12 my_site_name
    All releases can be found at https://www.drupal.org/project/drupal/releases → . For the latest 9.3 version, use drupal/recommended-project:^9.3 in the above command, or drupal/recommended-project:^9 for latest Drupal 9 version.

    Installing with composer via docker
    You might be using a docker based workflow and therefore do not have composer and its dependencies already installed. If this is the case then you can create the drupal project with this command:

    docker run --rm -i --tty -v $PWD:/app composer create-project drupal/recommended-project my_site_name --ignore-platform-reqs
    Install Drupal using the standard web interface
    After composer finishes downloading the packages, you can navigate your browser to your site's url and start the setup. It'll ask for the database credentials, a name for the admin user and some basic information.

    Install Drupal using the command line
    You can use Drush to install Drupal from the command line. Add Drush in your project by running: composer require drush/drush and use drush site:install to run the command line setup wizard. Without any arguments it'll install the standard profile and ask only for database credentials.

    drush site:install
    If you are evaluating Drupal and just want to see a site, you might try the quickstart feature. After creating the project you could type php ./web/core/scripts/drupal quick-start demo_umami and see a demo site.

    To do a modified install
    If you want to modify some of the properties of the downloaded composer.json before composer install is executed, use the --no-install flag when running composer create-project. For example, it is possible that you want to rename the subdirectory 'web' to something else.

    To do that:

    Run composer create-project --no-install drupal/recommended-project my_site_name
    Change directories to my_site_name and edit the composer.json file to suit your needs. For example, to change the sub-directory from 'web' to something else, the keys to modify are the 'extra' sub-keys 'web-root' (under 'drupal-scaffold') and 'installer-paths'.
    Run composer install to download Drupal and all its dependencies.
    If you are used to building Drupal site via Drush make, refer to the FAQs in Drupal's Composer template documentation to learn the difference between this option and Drush make.

    Download contributed modules, themes and their dependencies using Composer
    It is increasingly common that contributed Drupal modules have dependencies to third party libraries. Some of these modules can only be installed using Composer. If you downloaded Drupal core using Composer, or if one of your modules requires Composer, you should also use Composer to download all modules and themes using Composer. Sites may encounter critical update problems if they mix Composer updates and other update methods.

    This chapter has the following sub-sections:

    Download contributed modules and themes using Composer
    Define the directories to which Drupal projects should be downloaded if needed
    Download contributed modules and themes using Composer
    To download contributed Drupal modules or themes with composer:

    Run composer require drupal/module_name
    For example: composer require drupal/token
    This needs to be executed at the root of your Drupal install but not at the same level as the core directory.
    Composer will then automatically update your composer.json, adding the module to all the other requirements in the list, like this:

    {
    "require": {
    "drupal/token": "^1.5"
    }
    }
    Composer will download the module and all the possible dependencies it may have.

    You can enable the Drupal module in two ways:

    By using the standard Drupal web-browser interface.
    Using a command line tool like Drush or Drupal Console - see Installing Modules from the Command Line.
    You can use either the project name, or the specific module name within a project when requiring modules:

    Composer will download the whole project that contains a particular module.
    For example, if you need the fe_block module from the features_extra project you can do either from the following options:
    composer require drupal/features_extra
    composer require drupal/fe_block
    Specifying a version
    You can specify the version of the module / theme you want to download as follows:

    composer require 'drupal/module_name:version'
    (...where version is replaced with a Composer version or version constraint.)

    For example:

    composer require 'drupal/token:^1.5'
    composer require 'drupal/simple_fb_connect:~3.0'
    composer require 'drupal/ctools:3.0.0-alpha26'
    composer require 'drupal/token:1.x-dev'
    To avoid problems on different terminals/shells, surround the version in quotes as in the examples above.

    Note!: On Windows, the (single) quotes actually might break the version specification and lead to failure to install with 'Could not parse version constraint .....': Invalid version string "....'".

    Without the quotes, e.g.:

    composer require drupal/ctools:^3.7
    or with double quotes instead of single quotes, e.g.:

    composer require "drupal/ctools:^3.7"
    it works well. See Comment 'Composer problems: "could not parse version constraint"'.

    In the above examples, the versions map as follows:

    ^1.5: maps to the latest stable 8.x-1.x release of the module.
    ~3.0: maps to the latest stable 8.x-3.x release of the module.
    3.0.0-alpha26: maps to version 8.x-3.0-alpha26
    1.x-dev: maps to 8.x-1.x-dev
    For more on version constraints with ~ (tilde) and ^ (caret) see Next Significant Release Operators.

    It is recommended to indicate the version of the contributed module that you want to download. In the example above, Simple FB Connect can be updated to a later version of the 8.x-3.x branch but Composer will not automatically update to 8.x-4.x if it would be released.

    Using dev versions
    Development releases contain changes which their maintainers have not put into a tagged release. They may not be stable, may not have an upgrade path, and any security issues that are only present in dev releases can be disclosed publicly. It is strongly recommend that you use tagged releases of modules rather than development versions, however in some cases you may find it necessary to use a dev version.

    Because dev versions are pure git clones, they do not have additional version information provided by Drupal.org. This means update manager in Drupal core will not function properly for modules installed with dev versions.

    To resolve this, you can install the Composer Deploy project.

    About semantic versioning
    Drupal.org contributed projects are currently not versioned with true semantic versioning. However, the Composer service on Drupal.org translates the Contrib project version schema into a semver format that Composer can understand. This 'semver shim' will also allow Drupal.org to be flexible if the versioning standard for Contrib changes. The key element to this shim is that each major version of Drupal Core has a separate endpoint, to prevent collisions between the D7 version of a module and the D8 version

    Example D7 Endpoint Version Mapping
    Current Format Translated Format
    {Platform.x}-{major}.{minor}-{stability#} {major}.{minor}.0-{stability}{#}
    7.x-3.4-beta2 3.4.0-beta2
    7.x-2.10-rc2 2.10.0-rc2
    7.x-1.0-unstable3 unstable releases will not be translated, and not available to composer
    7.x-1.0-alpha5 1.0.0-alpha5
    7.x-0.1-rc2 0.1.0-rc2
    7.x-1.x-dev 1.x-dev
    Using Composer search
    Drupal.org's composer endpoints for Drupal 7 through 10 all support the Composer search function - so you can also search for Drupal projects from the command line. The format for using Composer search is:

    composer search views
    Using Composer browse
    Drupal.org's composer endpoints for Drupal 7 through 10 all support the Composer browse function - so you can find additional information about Drupal projects from the command line. The format for using Composer browse is:

    composer browse drupal/token
    Define the directories to which modules, themes, libraries, etc. should be downloaded
    If you want to change the locations in the file system where packages are installed, you can modify the "installer-paths" section of the composer.json file. This also can be useful if you need to have specific packages installed in their own locations.

    This approach uses the composer/installers package and uses configuration such as this:

    "extra": {
    "installer-paths": {
    "core": ["type:drupal-core"],
    "libraries/{$name}": ["type:drupal-library"],
    "modules/contrib/{$name}": ["type:drupal-module"],
    "profiles/contrib/{$name}": ["type:drupal-profile"],
    "themes/contrib/{$name}": ["type:drupal-theme"],
    "drush/{$name}": ["type:drupal-drush"],
    "modules/custom/{$name}": ["type:drupal-custom-module"],
    "themes/custom/{$name}": ["type:drupal-custom-theme"]
    }
    }
    Note: Custom modules and themes paths requires composer/installers package v1.0.24 and up.

    In addition to the package type-based installation locations you can use vendor specific ones, like this:

    "extra": {
    "installer-paths": {
    "web/libraries/ckeditor/plugins/{$name}": ["vendor:ckeditor-plugin"]
    }
    }
    Or package specific ones, like this:

    "extra": {
    "installer-paths": {
    "web/libraries/{$name}": [ "enyo/dropzone" ]
    }
    }
    Note: If a particular package matches multiple installer-paths entries, the first one that matches will be used.

  • Comment 9 months ago →
    🇨🇦Canada Jaypan

    Quoting from the link, it is right there on the page:

    You could have saved us both some time had you done that in the first place.

  • Status changed to Postponed: needs info 9 months ago
  • Comment 9 months ago →
    🇬🇧United Kingdom catch

    No you could have read the documentation provided before typing out two full paragraphs that were incorrect. Given that's been settled, moving this to more info.

  • Comment 9 months ago →
    🇨🇦Canada Jaypan

    No you could have read the documentation provided before typing out two full paragraphs that were incorrect.

    I read it, and did not find the section you vaguely referred to. This one is on you mate.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024