Move StreamWrappers from strings to URL objects

Created on 25 July 2023, over 1 year ago
Updated 11 January 2024, 10 months ago

Problem/Motivation

StreamWrapper's currently deal with static strings for URLs which means they're not opted in to any cacheability information system, even though URLs can have cacheability information as outlined in 🌱 [Meta] Allow StreamWrapper's to provide cacheability metadata Active .

Additionally most StreamWrappers internally already work with a Url which is then converted to a string, so we have the information but are throwing it away.

This issue probably also partially fixes πŸ› file_create_url() causes LogicException when used with certain stream wrappers Needs work or at least has a conflicting solution: the referenced issue implements getUrl as a protected method.

Steps to reproduce

Proposed resolution

Introduce getUrl to StreamWrapper so that it provides normal URL objects which contain routing information that can be provided with cacheability information. Additionally this benefits from improvements when URL objects themselves get cacheability information (e.g. because they're an external time-restricted link).

We specifically do not use GeneratedUrl because it contains less information than a Url object, specifically we make assumptions about how a URL should be consumed (absolute/relative) which other systems can better judge. We can see this in action by the FileUrlGenerator breaking apart output of the stream wrapper and converting it to a Url itself.

Remaining tasks

In a follow-up that will land in a next major version:

  • Deprecate StreamWrapperGetUrlInterface
  • Implement getUrl and extend StreamWrapperGetUrlInterface in StreamWrapperInterface

In a follow-up that will land in a next major version + 1:

  • Remove StreamWrapperGetUrlInterface

User interface changes

API changes

  • Introduce StreamWrapperGetUrlInterface which contains the getUrl method.
  • All concrete StreamWrappers in Drupal core implement StreamWrapperGetUrlInterface
  • Deprecate StreamWrapperInterface::getExternalUrl
  • FileUrlGenerator throws a deprecation error in case StreamWrapperGetUrlInterface is not implemented for a StreamWrapperInterface implementation

Data model changes

Release notes snippet

✨ Feature request
Status

Needs work

Version

11.0 πŸ”₯

Component
File systemΒ  β†’

Last updated about 11 hours ago

Created by

πŸ‡³πŸ‡±Netherlands kingdutch

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @kingdutch
  • Issue was unassigned.
  • Status changed to Needs review over 1 year ago
  • last update over 1 year ago
    Custom Commands Failed
  • πŸ‡³πŸ‡±Netherlands kingdutch
  • Status changed to Needs work over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    for the CC failure.

    Also should the test coverage be expanded a little more?

  • Status changed to Needs review over 1 year ago
  • last update over 1 year ago
    Custom Commands Failed
  • πŸ‡³πŸ‡±Netherlands kingdutch
  • πŸ‡³πŸ‡±Netherlands kingdutch

    Also should the test coverage be expanded a little more?

    I can't find any current getExternalUrl test coverage to migrate. My personal thoughts was that since this was used by FileUrlGenerator which has explicit tests for the path using specific stream wrappers, that that was enough test coverage. The need not to edit existing tests (assuming they pass; with the exception of the test streamwrapper implementation) would prove that the change in this low-level system works as intended.

  • last update over 1 year ago
    29,864 pass, 8 fail
  • πŸ‡³πŸ‡±Netherlands kingdutch
  • Status changed to Needs work over 1 year ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland
    +++ b/core/lib/Drupal/Core/StreamWrapper/PublicStream.php
    @@ -47,8 +47,19 @@ public function getDirectoryPath() {
    +   */
    +  public function getUrl() : Url {
         $path = str_replace('\\', '/', $this->getTarget());
    -    return static::baseUrl() . '/' . UrlHelper::encodePath($path);
    +    // We must replace the base URL with "base:" so that link modification works correctly
    +    // downstream, but we can't change static::baseUrl because it might be relied on by others.
    +    $base_url = str_replace($GLOBALS['base_url'], "base:", static::baseUrl());
    +    return Url::fromUri($base_url . '/' . $path);
       }
    

    There's an encoding issue somewhere based on some of the test fails (at least for kernel tests), as well as issues with index.php in the path.

    baseUrl() has support for a file_public_base_url setting, this will likely break that in some cases (\Drupal\Tests\file\Kernel\FileUrlTest::testFilesUrlWithDifferentHostName) doesn't fail, but I think that's because it doesn't contain the base url in it. If it would, it would be completely messed up with this and we should extend the test with such a case IMHO.

    I think we should inline baseUrl() and possibly deprecate it here if there's no other use.

  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    One more thing to consider here. We might want to deprecate getExternalUrl() only in D11 for D12 as we deprecate the separate interface. Modules that will want to remain compatible with D10 and D11 will need to keep the method anyway and that's going to be confusing otherwise.

  • First commit to issue fork.
  • Merge request !5740Resolve #3376892 "Move streamwrappers from" β†’ (Open) created by Taran2L
  • πŸ‡ΊπŸ‡¦Ukraine Taran2L Lviv

    Taran2L β†’ changed the visibility of the branch 3376892-move-streamwrappers-from to hidden.

  • πŸ‡ΊπŸ‡¦Ukraine Taran2L Lviv

    Taran2L β†’ changed the visibility of the branch 3376892-move-streamwrappers-from to active.

  • πŸ‡ΊπŸ‡¦Ukraine Taran2L Lviv

    ..., as well as issues with index.php in the path.

    This was fixed by forcing skipping the script part as it is a URL to a file and not a route.

    baseUrl() has support for a file_public_base_url setting, this will likely break that in some cases (\Drupal\Tests\file\Kernel\FileUrlTest::testFilesUrlWithDifferentHostName) doesn't fail, but I think that's because it doesn't contain the base url in it. If it would, it would be completely messed up with this and we should extend the test with such a case IMHO.

    Actually, it was working as before, except encoding part was missing, fixed

    I think we should inline baseUrl() and possibly deprecate it here if there's no other use.

    I would, as @Kingdutch suggested, add all deprecations in the follow-up

    There's an encoding issue somewhere based on some of the test fails (at least for kernel tests)

    Huh, this one is very interesting. Tests are checking whether generated URLs match expected value, but the value itself is "artificial", as in browser context URL like /vfs://abcde.txt does not make any sense, i.e. browser cannot act = download these anyways.

    The issues is that base:/vfs://abcde.txt is being double encoded, as Url class treats /vfs://abcde.txt as path and encodes it, right

    When using it without base:, there is a malformed URI, as /vfs://abcde.txt is not valid

    And, finally vfs://abcde.txt does not work, as vfs:// is not in allowed protocols list.

    So, I think those tests needs to be changed a bit, for example by allowing vfs:// protocol as a valid one for testing, and then it all would work properly

  • πŸ‡ΊπŸ‡¦Ukraine Taran2L Lviv

    Made it pass all the tests, but vfs:// thing is not cool. Probably there should be a core wrapper around vfs stream wrapper, so it just works without those conditions and special treatment

  • Status changed to Needs review 12 months ago
  • πŸ‡ΊπŸ‡¦Ukraine Taran2L Lviv
  • Status changed to Needs work 10 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Seems to need manual rebasing.

Production build 0.71.5 2024