[regression] FTPExtension class can no longer connect as of 9.5.x

Created on 28 February 2023, over 1 year ago
Updated 19 June 2024, 5 months ago

Problem/Motivation

πŸ› Fix magic connection property access from \Drupal\Core\FileTransfer\FileTransfer::__get() Fixed Introduced magic methods to access Drupal\Core\FileTransfer\FTPExtension->connection however now FTPExtension->connect() always throws an exception (which is caught) due to nested calls to __get() .

Error:
Notice: Undefined property: Drupal\Core\FileTransfer\FTPExtension::$connection in Drupal\Core\FileTransfer\FTPExtension->connect() (line 16 of core/lib/Drupal/Core/FileTransfer/FTPExtension.php).

Cannot connect to FTP Server, check settings

Why:
1. Call FTPExtension->connect(); in code.
2. FTPExtension->connect() sets the $connection property
3. $connection is set via parent class FileTransfer's private property $connectionHandle via __set()
4. FTPExtension->connect() then checks if $connection property was set to non-Falsey value.
5. This triggers FileTransfer's __get() which calls FTPExtension->connect()
4. Repeat step 2 ( FTPExtension->connect() sets the $connection property)
5. Repeat step 3 ( $connection is set via parent class FileTransfer's private property $connectionHandle via __set() )
6. Repeat step 4 ( FTPExtension->connect() then checks if $connection property was set to non-Falsey value.)
7. FileTransfer's __get() is not triggered as __get() has already been called in the call stack with the same parameters and so PHP treats it as regular property which is undefined (since the real property is stored in private property $connectionHandle ), so the else condition is triggered which is throwing an exception. (If it didn't it, would be an infinite loop which is equally bad)

Supporting information regarding Zend PHP on nested magic methods.
@see https://github.com/facebook/hhvm/issues/1312

For reference, the PHP behaviour allows calls to magic methods to recurse as long as there isn't an identical call on the stack, this means you can still trigger a magic method from within a magic method as long as the call itself is different.

Steps to reproduce

How to throw the error in custom code:

use Drupal\Core\FileTransfer\FileTransferException;

$data = NULL;
$url = 'ftp://ftp.myserver.com/';
$info = drupal_get_filetransfer_info();
$class = $info['ftp']['class'];
$ftp = $class::factory(DRUPAL_ROOT, [
  'advanced' => [
    'hostname' => parse_url($url, PHP_URL_HOST),
  ],
  'username' => 'anonymous',
  'port' => 21,
]);
try {
  // Login, throws exception if a login error, else defines $ftp->connection.
  // Exceptio nis thrown here.
  $ftp->connect();

  // CODE DOES NOT GET TO THIS POINT, demo purposes to get file content.
  ftp_pasv($ftp->connection, TRUE);
  ob_start();
  if (ftp_get($ftp->connection, 'php://output', parse_url($url, PHP_URL_PATH), FTP_BINARY)) {
    $data = ob_get_contents();
    ob_end_clean();
  }
  ftp_close($ftp->connection);
}
catch (FileTransferException $e) {
  \Drupal::logger()->error($e->getMessage());
  \Drupal::messenger()->addError($e->getMessage());
}

// Echo file data if successful.
if ($data) {
  echo $data;
}

Mock code showing the issue:

class PropertyTest
{
  /**  Location for overloaded data.  */
  private $myPrivate;
  public function __set($name, $value) {
    if ($name == 'myVar') {
    echo "Setting 'myVar' to '$value'\n";
    $this->myPrivate = $value;
    }
    else {
    echo "SET FAILED\n";  
    }
  }
  public function __get($name){
    echo "Getting 'myVar'\n";
    $this->connect();
    return ($name == 'myVar') ? $this->myPrivate : 'GET FAILED';
  }
}

class PropertyTestchild extends PropertyTest {
  public function connect() {
  $this->myVar = 'TEST';
  echo "Parent connect run\n";
  if (!$this->myVar) {
    echo "This property 'myVar' was not set?!\n";
  }
  }
}
echo "<pre>\n";
$obj = new PropertyTestchild();
$obj->connect();
echo "</pre>\n";

Output:

Setting 'myVar' to 'TEST'
Parent connect run
Getting 'myVar'
Setting 'myVar' to 'TEST'
Parent connect run

Warning: Undefined property: PropertyTestchild::$myVar in /home/user/scripts/code.php on line 26
This property 'myVar' was not set?!

Proposed resolution

Only run `connect()` once or revert to previous commit.

One run once dummy code:

class PropertyTest
{
  /**  Location for overloaded data.  */
  private $myPrivate;
  public function __set($name, $value) {
    if ($name == 'myVar') {
    echo "Setting 'myVar' to '$value'\n";
    $this->myPrivate = $value;
    }
    else {
    echo "SET FAILED\n";  
    }
  }
  public function __get($name){
    echo "Getting 'data'\n";
    $this->connect();
    return ($name == 'myVar') ? $this->myPrivate : 'GET FAILED';
  }
}

class PropertyTestchild extends PropertyTest {
  private $hasConnectRun = FALSE;
  public function connect() {
  if ($this->hasConnectRun) {
      return;
    }
    $this->hasConnectRun = TRUE;
  $this->myVar = 'TEST';
  echo "Parent connect run\n";
  if (!$this->myVar) {
    echo "This property 'myVar' was not set?!\n";
  }
  }
}
echo "<pre>\n";
$obj = new PropertyTestchild();
$obj->connect();
echo "</pre>\n";

Output:

Setting 'myVar' to 'TEST'
Parent connect run
Getting 'data'

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

πŸ› Bug report
Status

Fixed

Version

10.2 ✨

Component
File systemΒ  β†’

Last updated about 11 hours ago

Created by

πŸ‡¦πŸ‡ΊAustralia silverham

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Issue created by @silverham
  • πŸ‡¦πŸ‡ΊAustralia silverham

    Patch based on running FTPExtension->connect() ; once.

  • Status changed to Needs review over 1 year ago
  • πŸ‡«πŸ‡·France andypost
    +++ b/core/lib/Drupal/Core/FileTransfer/FTPExtension.php
    @@ -11,6 +11,10 @@ class FTPExtension extends FTP implements ChmodInterface {
    +    // Ensure `$this->connect()` is only called once.
    +    if ($this->setConnectRunState()) {
    
    +++ b/core/lib/Drupal/Core/FileTransfer/FileTransfer.php
    @@ -62,6 +62,13 @@ abstract class FileTransfer {
    +  private $hasConnectRunState = FALSE;
    
    @@ -439,6 +452,19 @@ public function setChroot() {
    +  public function setConnectRunState() {
    +    $hasConnectRunState = $this->hasConnectRunState;
    +    $this->hasConnectRunState = TRUE;
    +    return $hasConnectRunState;
    
    +++ b/core/lib/Drupal/Core/FileTransfer/SSH.php
    @@ -22,6 +22,10 @@ public function __construct($jail, $username, $password, $hostname = "localhost"
    +    // Ensure `$this->connect()` is only called once.
    +    if ($this->setConnectRunState()) {
    

    I think there's $connectionHandle private variable which could be used instead

  • πŸ‡¦πŸ‡ΊAustralia silverham

    Interesting, I see that ftp_connect() and ssh2_connect() both return a non-NULL value, so FALSE or a resource object so an isset() check could work. I can redo patch later, or feel free to do yourself if you get to it first, no worries.

  • πŸ‡¦πŸ‡ΊAustralia silverham

    Updated patch to use a do a null check as per suggested : isset($this->connection) .

    Didn't use isset($this->connectionHandle) in subclass is it is private (no subclass access to parent) and all sub-classes still use naming convention of $this->connection

  • Status changed to Needs work over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    This issue is being reviewed by the kind folks in Slack, #needs-review-queue-initiative. We are working to keep the size of Needs Review queue under control, following Review a patch or merge request β†’ as a guide.

    Next step would be to add a test case to show the issue.

  • Hi,
    As one of the few remaining users who can't use composer, I was somewhat interested in getting this fixed.
    So if I gave you a working solution, would anybody be willing to implement it?

    What I did was (in FileTransfer.php)

    • Kept the added property declarations but renamed connection back (removed -Handle) and made it public
    • Added code to make sure chrootHandle is always set when needed (that part broke with the addition of the declaration) and fixed some leftover uses of just chroot (without 'Path')

    Im adding adding this as a .zip file, I know that's not the right way but learning how to create a merge request seems like a lot of work...

    Btw, I tested this on a site with shared hosting with a chrroted FTP. I also verified that the 'Local' subclass still works.

    Some more Details on what I dit, and why:
    The change made with Drupal 9.5 was to add property declarations for $connection, $croot and $jail.
    connection and chroot were also renamed and made private respectively protected.
    Some more 'magic' were function added, probably to make it (unit tests?) work again.

    Renaming connection was imho ill-advised. Because that means the __get() has two different things to do: providing access from the outside and the FTPExension subclass, and connecting when $connection is absent in the class itself.That just leads to unnecessary complications.

    By the way, connection is deliberatley unset in core\modules\update\update.authorize.inc
    See "unset($filetransfer->connection);" (with an explanation added)

    So what happens is: From the form where the ftp credentials are entered, a connection is established, but probably just to test it.
    When we come back wih a task, connection is therefore completely gone. Also, chrootPath has somehow lost it's value.
    But because it is declared, it is null, not absent. So while connection reestablishes itself, the 'magic' __get is nottriggered by it. Apparently, null is not unaccessible for the sake of this.

    So I had to find a location where to call setChroot() and found it in fixRemotePath().

    After that, all the other magic functions were no longer needed.

  • πŸ‡¬πŸ‡§United Kingdom SoulReceiver

    The latest patch works wonderfully. Tested in Drupal 10.2.5, it fixed the issue for me with the SSH library. Beforehand I was too was getting that undefined `connection` property, but this patch resolves that and eliminates the recursive behaviour that would lead to an infinite loop if the exception weren't being thrown.

  • Status changed to RTBC 7 months ago
  • πŸ‡¬πŸ‡§United Kingdom longwave UK

    Unsure how we can add a test for this, this class doesn't have any existing test coverage and without the ssh2 extension or an FTP server to connect to I'm not sure what value any test would have.

    The fix and explanation make sense to me, the additional comments in this (untested) code are welcome too, going to mark this as RTBC. Although it does also point to the fact that very few people probably use this if it's been broken since 9.5...

  • First commit to issue fork.
  • Status changed to Needs work 7 months ago
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    This is testable and the change in #3295821: Ignore: patch testing issue for PHP 8.2 attributes β†’ was implemented incorrectly. We can fix this without changing the logic of things that extend \Drupal\Core\FileTransfer\FileTransfer. In fact the changes to \Drupal\Tests\system\Functional\FileTransfer\TestFileTransfer in #3295821: Ignore: patch testing issue for PHP 8.2 attributes β†’ should have warned us that we were doing this wrong.

  • Status changed to Needs review 7 months ago
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ
  • Merge request !7905Fix FileTransfer β†’ (Closed) created by alexpott
  • Pipeline finished with Failed
    7 months ago
    Total: 452s
    #163860
  • Pipeline finished with Failed
    7 months ago
    Total: 914s
    #163872
  • Status changed to Needs work 7 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Appears to have a test failure

        1) Drupal\Tests\system\Functional\FileTransfer\FileTransferTest::testJail
        Error: Call to a member function run() on null
        
        /builds/project/drupal/core/modules/system/tests/src/Functional/FileTransfer/TestFileTransfer.php:62
        /builds/project/drupal/core/lib/Drupal/Core/FileTransfer/FileTransfer.php:216
        /builds/project/drupal/core/lib/Drupal/Core/FileTransfer/FileTransfer.php:335
        /builds/project/drupal/core/lib/Drupal/Core/FileTransfer/FileTransfer.php:180
        /builds/project/drupal/core/modules/system/tests/src/Functional/FileTransfer/FileTransferTest.php:95
        /builds/project/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:729
    
  • Pipeline finished with Canceled
    7 months ago
    Total: 140s
    #166459
  • Status changed to Needs review 7 months ago
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ
  • Status changed to RTBC 7 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Hiding patches for clarity

    Tests are all green so switch to chroot and connection didn't break anything.

    Not sure how else to test but seems fine to me.

  • Pipeline finished with Success
    7 months ago
    Total: 3703s
    #166463
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    Added a comment that shows we have coverage of the problem in the issue summary by reverting a change to the test code that was made in the earlier issue.

    Also the test only job fails as expected - https://git.drupalcode.org/project/drupal/-/jobs/1534956

  • First commit to issue fork.
  • Status changed to Needs work 6 months ago
  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    This one needs a reroll for phpstan baseline folks. Thanks

  • πŸ‡¦πŸ‡ΊAustralia kim.pepper πŸ„β€β™‚οΈπŸ‡¦πŸ‡ΊSydney, Australia

    kim.pepper β†’ made their first commit to this issue’s fork.

  • Status changed to Needs review 6 months ago
  • πŸ‡¦πŸ‡ΊAustralia kim.pepper πŸ„β€β™‚οΈπŸ‡¦πŸ‡ΊSydney, Australia

    Rebased on 11.x

  • Pipeline finished with Failed
    6 months ago
    Total: 875s
    #176615
  • Pipeline finished with Success
    6 months ago
    Total: 570s
    #176631
  • Pipeline finished with Failed
    6 months ago
    Total: 633s
    #176961
  • Pipeline finished with Canceled
    6 months ago
    Total: 128s
    #176993
  • Status changed to RTBC 6 months ago
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    I thought we could remove the #[\AllowDynamicProperties] because we had the @property tags - it's good enough for PHPStan but not for PHP itself... so we still need that attribute.

    Setting back to rtbc because @kim.pepper's fix was only phpstan and comments.

  • Pipeline finished with Success
    6 months ago
    Total: 573s
    #176995
    • larowlan β†’ committed bf153748 on 10.3.x
      Issue #3344877 by alexpott, silverham: [regression] FTPExtension class...
    • larowlan β†’ committed 4e04b15a on 10.4.x
      Issue #3344877 by alexpott, silverham: [regression] FTPExtension class...
    • larowlan β†’ committed 3843b614 on 11.0.x
      Issue #3344877 by alexpott, silverham: [regression] FTPExtension class...
    • larowlan β†’ committed 6284267b on 11.x
      Issue #3344877 by alexpott, silverham: [regression] FTPExtension class...
  • Status changed to Downport 6 months ago
  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Committed to 11.x, 11.0.x, 10.4.x and 10.3.x

    Doesn't apply to 10.2.x - moving to there for a possible backport

  • First commit to issue fork.
  • Status changed to RTBC 6 months ago
  • πŸ‡¬πŸ‡§United Kingdom longwave UK

    Same changes locally; this should also fix the PHPStan fails in 10.2.x at https://git.drupalcode.org/project/drupal/-/pipelines/191534

  • Status changed to Downport 6 months ago
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    That's a nice benefit. There were conflicts in the baseline so I rebuilt it.

  • Status changed to RTBC 6 months ago
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    I did not intend to change the status

  • Status changed to Fixed 6 months ago
  • πŸ‡¬πŸ‡§United Kingdom longwave UK

    Committed as this is just a backport and also fixes PHPStan in time for a 10.2.x release this week.

    Committed c141f9e and pushed to 10.2.x. Thanks!

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024