Add token support for Http Fetcher source url

Hi,

Needed this on a personal project so custom coded it very roughly/quickly. It's a bit rough coding standards-wise & mashed into a bigger custom module but here's some code snippets that are hopefully useful for someone else (a near clone of the feeds HTTPFetcher fetcher):

/src/Feeds/Fetcher/Form/HttpTokensFetcherFeedForm.php:

<?php

namespace Drupal\MODULE_NAME\Feeds\Fetcher\Form;

use Drupal\Core\Form\FormStateInterface;
use Drupal\feeds\FeedInterface;
use Drupal\feeds\Feeds\Fetcher\Form\HttpFetcherFeedForm;

/**
 * Provides a form on the feed edit page for the HttpFetcher.
 */
class HttpTokensFetcherFeedForm extends HttpFetcherFeedForm {

  /**
   * {@inheritdoc}
   */
  public function buildConfigurationForm(array $form, FormStateInterface $form_state, FeedInterface $feed = NULL) {
    parent::buildConfigurationForm($form, $form_state, $feed);

    $form['source'] = [
      '#title' => $this->t('Feed URL'),
      '#type' => 'url',
      '#default_value' => rawurldecode($feed->getSource()), // maintains token in source without encoding
      '#maxlength' => 2048,
      '#required' => TRUE,
    ];

    // Global tokens
    $form['tokens'] = [
      '#theme' => 'token_tree_link',
      '#token_types' => [
        'global',
      ],
    ];   

    return $form;
  }
}

/src/Feeds/Fetcher/HttpTokensFetcher.php:

<?php

namespace Drupal\MODULE_NAME\Feeds\Fetcher;


use Drupal\feeds\Exception\EmptyFeedException;
use Drupal\feeds\Feeds\Fetcher\HttpFetcher;
use Drupal\feeds\Result\HttpFetcherResult;
use Drupal\feeds\StateInterface;
use Drupal\feeds\Utility\Feed;
use GuzzleHttp\Exception\RequestException;
use GuzzleHttp\RequestOptions;
use Symfony\Component\HttpFoundation\Response;
use Drupal\feeds\FeedInterface;

/**
 * Defines an HTTP fetcher.
 *
 * @FeedsFetcher(
 *   id = "http_tokens_fetcher",
 *   title = @Translation("Download URL with tokens"),
 *   description = @Translation("Downloads data from a URL using Drupal's HTTP request handler with tokens."),
 *   form = {
 *     "configuration" = "Drupal\feeds\Feeds\Fetcher\Form\HttpFetcherForm",
 *     "feed" = "Drupal\MODULE_NAME\Feeds\Fetcher\Form\HttpTokensFetcherFeedForm"
 *   }
 * )
 */
class HttpTokensFetcher extends HTTPFetcher {

  /**
   * {@inheritdoc}
   */
  public function fetch(FeedInterface $feed, StateInterface $state) {
    $sink = $this->fileSystem->tempnam('temporary://', 'feeds_http_fetcher');
    $sink = $this->fileSystem->realpath($sink);

    $source_url_encoded = rawurldecode($feed->getSource()); // @todo - Drupal way for this?

    // Global tokens
    $token_service = \Drupal::token(); // @todo - inject this
    $source_url = $token_service->replace($source_url_encoded);

    // Get cache key if caching is enabled.
    $cache_key = $this->useCache() ? $this->getCacheKey($feed) : FALSE;

    $response = $this->get($source_url, $sink, $cache_key);
    // @todo Handle redirects.
    // @codingStandardsIgnoreStart
    // $feed->setSource($response->getEffectiveUrl());
    // @codingStandardsIgnoreEnd

    // 304, nothing to see here.
    if ($response->getStatusCode() == Response::HTTP_NOT_MODIFIED) {
      $state->setMessage($this->t('The feed has not been updated.'));
      throw new EmptyFeedException();
    }

    return new HttpFetcherResult($sink, $response->getHeaders());
  }
}

Cheers
Tom

Thank you Tom!
I still need to test this, but I wonder what the module maintainer Chriz thinks of this and if it could be added to Feeds module codebase..

Something like #12 could work. 🙂

It does need work:

• Dependency injection
• Automated tests
• Permission to use tokens?

And I wonder that when tokens are used in a url, if the url would still pass validation? 🤔 I guess it is going to fail validation if the field only consists of a token.
So perhaps the "Feed URL" should then become a regular plain text field. But that would mean though that if tokens are not used in the field, it still requires to be validated as url.

I mark this as "Needs work", even though there is no patch available yet. But there is code available.