[policy, no patch] Make PHP's OpenSSL extension a requirement for installing and using Package Manager (and therefore, Automatic Updates and Project Browser)

Tagging for release management and product management review. Since I already wrote my recommendation in the proposed resolution, I'm not tagging it for framework management review :)

I think based on how many packages are only available from https URLs, this shows that people using composer have the openssl extension locally. The unknown is how many hosting environments (which in many cases won't be running composer) have the openssl extension, but even if they don't, I don't think that overrides how much is only accessible on https URLs anyway. So I think we should just require the openssl extension. We might end up getting issues opened about it, but also we might not.

No input just moving to plan category. Maybe after a decision is made it can be a task again with a patch?

Given #5, marking this as RTBC and untagging "Needs release manager review". Also updated the title and issue summary accordingly.

+1 to adding the requirement. Having to add some kind of proxy infrastructure feels like a suboptimal solution and hopefully it won't come to that, and certainly I don't think we should be building that now. So let's add the requirement and see how we get on, and revisit if it does turn out to be a problem for some environments.

Assuming this issue is decided as written, the next question becomes: 🌱 [policy, no patch] Disallow using Package Manager (and therefore Automatic Updates and Project Browser) when Composer's disable-tls setting is true Needs review .

FWIW I am not as confident as @catch about this issue, and it's problematic when folks remove a tag for a role they do not have. If we mean to untag it, we will.

I would really want some industry data before making this decision, especially from LTS-inclined sectors like government or higher ed.

To expand on #7 this is the central bit of the issue summary for me:

As far as I can tell, api.github.com does not support http requests. For example, the response to curl -v http://api.github.com/repos/symfony/http-foundation/zipball/718a97ed430d... (the zip containing the 6.3.0 release of symfony/http-foundation) is a 301 redirect to the https URL. Therefore, on sites without the openssl extension, Package Manager would be unable to download any packages from GitHub, which is where Symfony and other Drupal dependencies are hosted. In order for Package Manager to work on site without the openssl extension, we could in theory run a proxy on drupal.org to serve GitHub's files over http, but it's questionable whether operating such a proxy is a good idea.

If we don't require openssl, it doesn't mean that package manager will work without it, it means that it will fail on not being able to download resources which are already only available over https, and not available over http.

This means we have three choices:
1. Warn people up-front that they need the openssl extension if they don't have it.
2. Don't warn people and then them run into errors when they try to use it
3. Circumvent SSL for all of Drupal's dependencies by proxying them on Drupal.org, requiring significant engineering effort from the DA and relying entirely on TUF for security.

When the original discussion about this were had, it was pre Let's Encrypt etc., and it's quite likely #2 would have been 'people will get stuff over http instead, and code on Drupal.org will be protected by TUF, and we'd try to get TUF adopted elsewhere', but unless there's hidden non-SSL endpoints for github around somewhere, that doesn't seem like it's an option any more, which leaves #2 and #3 for me as considerably worse options than #1.

FWIW GitHub made the decision to go HTTPS only back in 2013, which means Composer users have needed openssl for ten years now: https://github.com/composer/composer/issues/1955

BTW, it's not just about HTTPS versus no HTTPS. It's about weird, buggy, or problematic old versions of OpenSSL or TLS or whatever on older hosting stacks. At least, that's the risk AutoUpdates has been trying to mitigate since 2018. I'm going to ask David Strauss to comment on this issue since he was very concerned about this.

The issue summary says we have "no data to confirm or quantify the hypothesis" (re: missing or bugged SSL support). But is there no way to get such data? No sources from the internet beyond old threads? I agree that a modern technology stack is less likely to be affected, but there are a lot of dinosaur Drupal sites out there.

Barring existing sources, we could try constructing a survey or something. Or use the project messaging channel to ask folks to contact us if they have the versions we're trying to avoid, or no SSL?

Old and buggy SSL is more 🌱 [policy, no patch] Disallow using Package Manager (and therefore Automatic Updates and Project Browser) when Composer's disable-tls setting is true Needs review than this issue. Composer supports (and complains about) ignoring invalid SSL certificates, and we need to decide what to do about that too. But this issue is only about whether we try to support http-only requests which looks to be not possible without mirroring github.

Something that may create an issue with a few sites is LibreSSL, a fork of OpenSSL that removes older, insecure protocols, is lighter weight,and is preferred in some Linux distributions. Below is a link on configuring PHP7 for LibreSSL. IDK if it has gotten any easier in recent times.

Another problem to check is that certificate validation often doesn't check if it has been revoked.

https://stackoverflow.com/questions/43942717/build-php-7-using-libressl#...

I'm not sure how much we can be expected to babysit people with broken setups. While I've seen SSL root certificate store issues in the wild that would likely cause problems here, I've also seen systems with DNS resolver issues or IPv6 configuration issues that would also break Automatic Updates in a similar fashion - should we be trying to handle this too? There's lots of weird things going on especially inside corporate networks that we can't hope to cover up front.

The majority of sites are likely to be configured correctly, but I don't think we should aim to support ones that aren't, and to me that includes allowing HTTP downloads for the rare case that HTTPS is broken, given that GitHub themselves don't support this.

BTW, it's not just about HTTPS versus no HTTPS. It's about weird, buggy, or problematic old versions of OpenSSL or TLS or whatever on older hosting stacks. At least, that's the risk AutoUpdates has been trying to mitigate since 2018. I'm going to ask David Strauss to comment on this issue since he was historically very concerned about this.

xjm is correct about my concerns: we don't know the status of TLS client stacks on host systems. It's also hard to reason about them based on data we do have, too:

1. The TLS server stack for accessing a site may not be the same as the TLS client stack PHP may use for outbound requests. Even if a server uses the same TLS stack for server and client purposes, it being old/broken can be masked by something like throwing Cloudflare in front of the server. This issue affects ciphers and protocol versions.
2. A TLS stack that works fine for server purposes may fail for client purposes if the CA roots of trust on the client are missing or out of date. Let's Encrypt cross-signed for a while, but I believe they're now reliant on CA chains having their native root. This may also affect CAs like Fastly's emerging Certainly. We could ship and use our own CA data if we wanted to, but that has its own brittleness.
3. Even if we know the TLS client stack works properly for a specific connection Drupal makes, that only tells us that the stack is modern enough for that particular negotiation. Drupal.org requires TLS 1.2 and doesn't provide 1.3 at all, and it's not clear when >=1.3 will become a thing like >=1.2 did a few years ago.
4. TLS is sometimes messed with by nation states and corporate firewalls in ways that work fine for one purpose and break for another.

That all said, we've addressed my main concern with even our limited investment into TUF. If we find ourselves bitten by these issues, we have methods to isolate our root of trust and evaluation of the integrity of the chain.

My only question about requiring OpenSSL is whether it's the right requirement to ensure a (likely) functional client TLS stack. libcurl can leverage to other libraries like NSS; we have even used this in the past at Pantheon when Red Hat was trying to consolidate on NSS. Unless Composer leans on OpenSSL directly, libcurl will leverage whatever TLS it's built with.

Oh, and here's another wrinkle: Whether PHP is built with OpenSSL bindings is different from what libcurl is built with and using. It's possible to have libcurl be able to access HTTPS resources securely without OpenSSL directly linked into PHP at all. It's also possible for libcurl to be built with something like NSS and still have PHP linked to OpenSSL.

So, I would propose that the correct requirement here involves understanding how Composer uses HTTPS/TLS and requiring that. There's a good chance that the requirement we should have is that PHP's libcurl is built with TLS support -- and that should be actually more universal than whether PHP itself is linked with OpenSSL. Also, PHP can use its own bundled libcurl (IIRC, the default) or system libcurl (not the default but commonly used by distro packagers like in the Red Hat world), so any check of curl capabilities should use the PHP API and not shell out.

If we rely on composer that would mean skipping this issue and going directly to 🌱 [policy, no patch] Disallow using Package Manager (and therefore Automatic Updates and Project Browser) when Composer's disable-tls setting is true Needs review , but that also prevents accessing endpoints using self signed certificates etc. So is more restrictive than dropping http support.

My impression is that TLS client stacks seem to be in better shape now. There was a time when we got a handful of support requests and updating was the solution. I have not seen as many of those issues lately. That is completely un-scientific of course.

packages.drupal.org is only available over HTTPS and I expect it to stay that way. Requesting http://packages.drupal.org/8/composer.json redirects to HTTPS. TUF metadata will be served from that domain.

If there are good reasons to allow not using HTTPS, we can consider them. Offhand, I can’t remember the last time someone had a client stack issue, if any, with that domain.

@David Strauss: while I think Composer uses CURL for the actual downloads, and that CURL will use whatever SSL library it's compiled with, as far as I can tell Composer simply won't initiate any https requests without PHP's OpenSSL extension being enabled. I opened
https://github.com/composer/composer/discussions/11676 to ask if they've encountered a situation where someone compiles CURL with something other than OpenSSL and also disables (or doesn't enable) PHP's OpenSSL extension.

#20 was answered by @Seldaek (Composer maintainer). Copying here for easier scanning.

My question:

Composer flat out refuses to download from https URLs unless PHP's OpenSSL extension is enabled. Question for Composer maintainers: do you recall any support requests from people who want to use some other SSL library (e.g., NSS), and have compiled their CURL libraries accordingly, and don't have PHP's OpenSSL extension enabled but still have a system capable of making https requests through CURL? And if so, how have you handled those situations?

@Seldaek's answer:

Nope, nobody has ever asked as far as I can remember. FWIW, we do rely on openssl_* functions directly for CA bundle verification (in composer/ca-bundle), as well as in self-update to verify the phar file signature.

Re #17.2:

A TLS stack that works fine for server purposes may fail for client purposes if the CA roots of trust on the client are missing or out of date... We could ship and use our own CA data if we wanted to, but that has its own brittleness.

Composer already includes one from Mozilla that they keep up to date (see #3351190-18: [policy, no patch] Should Package Manager require Composer HTTPS? → ).

FWIW, we do rely on openssl_* functions directly for CA bundle verification (in composer/ca-bundle), as well as in self-update to verify the phar file signature.

So, leaning on my earlier suggestion to "require what will make Composer work," we would add/revise two requirements: (1) OpenSSL for PHP and (2) curl for PHP with functional TLS. I think we already require curl, so the latter might just be a small revision. Almost all curl is built with TLS, so "OpenSSL for PHP" is the real change here.

Composer already includes one from Mozilla that they keep up to date

Oh, interesting. Does it also fall back to the bundled CAs if the system CA trust roots are out of date (read: are missing essential roots of trust for key servers/endpoints)?

What is stopping this issue from going from RTBC to Fixed? Are "[policy, no patch] " issues marked "fixed"?

I want to know if I can make this change to the module or not

Based on the information we have now, it seems fine to require PHP's OpenSSL extension from product perspective. If it turns out that there's a large number of users impacted, we can always reassess this.

From UX perspective what is important is that we have clear messages when then requirements are not met, and documentation for how users may try to get make their environment compatible. I understand that we may not be able to detect all of the situations, or be aware of the different steps users may need to take. We can always try our best based on what we know and can do, and then improve this later on if we learn more.

I strongly feel that we don't have a choice here, and that we can always make things less restrictive later (infra challenges notwithstanding), but that making them more restrictive later would risk breaking existing sites. I've updated the issue summary with some more considerations.

I have been keeping up with this issue and although I have little experience with the details here I do support the the current proposal. The reasons put forward for this are convincing and I don't need to repeat what has already been said. And in general, I would rather Drupal been seen as striving to be secure than not. However, like xjm I would like data so we know how many Drupal sites will have problems but the explanations by @David Strauss convince me that too much is out of our control anyway.

Having said that there are more things to do if this is accepted, from #25 and doc updates. I added those to the proposed resolution

I've opened two issues to try to (imperfectly, see below) address this concern from @xjm:

I would really want some industry data before making this decision, especially from LTS-inclined sectors like government or higher ed.

The issue summary says we have "no data to confirm or quantify the hypothesis" (re: missing or bugged SSL support). But is there no way to get such data? No sources from the internet beyond old threads? I agree that a modern technology stack is less likely to be affected, but there are a lot of dinosaur Drupal sites out there.

Barring existing sources, we could try constructing a survey or something. Or use the project messaging channel to ask folks to contact us if they have the versions we're trying to avoid, or no SSL?

I'm concerned that if we were to rely on a survey, that would not give us enough information to be confident, especially about institutional hosts. The kinds of university or government departments that are already running Drupal 10 and are engaged with the community are more likely to run modern hosting stacks too. You need to keep up with PHP releases at a minimum starting from Drupal 8. The kinds of sites that might still be running Drupal 7 on PHP 7.3 are also less likely to respond to a survey, so we could get a false negative. We also have effectively zero sites getting project messaging messages because it's not stable/in standard yet. Even when it is, it would only cover up-to-date sites, until it's been in long enough for them to go out of date.

Additionally, unless we got an overwhelming number of responses, if one or two universities reply they're on Drupal 7 and PHP 7.3 with no openssl extension, that would also not necessarily be good data. When/if they update to Drupal 10, they might also do so on a more modern hosting stack with openssl compiled in, and then their initial report would be out of date. And if it's just a couple of places, that probably isn't enough to justify potentially months of Drupal Association infrastructure work up front, because people who are unaffected/not interested are less likely to respond too. What we probably will get (but hopefully not if the assumptions made here turn out to be true), is support requests/bug reports if people try to use automatic updates/package manager and can't on their environment, but that's a chicken and egg situation with it being beta in core to be able to produce a sufficient number or absence of those.

On top of that, I only in the past couple of weeks realised there's a stronger argument to default to requiring openssl and disable-tls: false to start with:

If we default to the stricter requirements first, we can loosen them later, and existing sites already relying on automatic updates would be unaffected.

However if we start off requiring neither openssl nor disable-tls: false, and 1000 sites rely on that, then the DA having set up an http TUF-enabled packagist/github proxy is for some reason not able to maintain it long term, then those 1000 sites, though we could warn them we're going to change things, if they take no action would end up stuck on a release they can't update from. It's much easier to choose or adapt hosting based on an up-front requirement than to have to migrate a production to new hosting or recompile PHP etc.

Given all of these things, and having talked also to @longwave and @quietone (but unfortunately not xjm in the past week or so), I'm going to go ahead and untag this issue for release manager review and mark this issue fixed. It is the only way to avoid potentially wasted up-front work and also is a decision we can much more easily reverse from this direction as more information comes in. Essentially an interim decision so we can have unambiguous requirements to start with, and revisit later.

The two issue I've opened are 🌱 Collect data on openssl and tls support across hosting environments Active and 🌱 [PP-*] [policy] Allow package manager to work without openssl and/or a valid certificate bundle Active .

Automatically closed - issue fixed for 2 weeks with no activity.