HTTP data fetcher plugin cannot have timeout assigned

Created on 11 March 2019, over 5 years ago
Updated 12 December 2023, 7 months ago

I am currently trying to work with a remote API that provides the data to a migration. The service is a bit slow and I trip the 30 second timeout and need to increase it. The HTTP data parser does not allow for this configuration override, it only allows for things within 'headers'.

It is illustrated in this method:

  /**
   * {@inheritdoc}
   */
  public function getResponse($url) {
    try {
      $options = ['headers' => $this->getRequestHeaders()];
      if (!empty($this->configuration['authentication'])) {
        $options = array_merge($options, $this->getAuthenticationPlugin()->getAuthenticationOptions());
      }
      $response = $this->httpClient->get($url, $options);
      if (empty($response)) {
        throw new MigrateException('No response at ' . $url . '.');
      }
    }
    catch (RequestException $e) {
      throw new MigrateException('Error message: ' . $e->getMessage() . ' at ' . $url . '.');
    }
    return $response;
  }

$options is set to a headers key, rendering any other options you want to set in the client moot.

$options = ['headers' => $this->getRequestHeaders(), 'timeout' => 90]; actually increases my timeout, but in its current state, there is no way to pass that along without declaring my own parser plugin and extending Http class to override this behavior.

I don't know if all client options need to be supported, but timeout seems like a fairly common thing to adjust.

For now I will need to extend the parser so I don't hack Migrate Plus, but it would be nice to see more options be supported.

πŸ› Bug report
Status

Closed: outdated

Version

4.1

Component

Plugins

Created by

πŸ‡ΊπŸ‡ΈUnited States kevinquillen

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡©πŸ‡°Denmark arnested

    Searching for how to set a timeout, I came across this issue.

    The issue is 5 years old, and I think is now outdated.

    We're on Migrate Plus version 6.0.1, and we can set request options for the HTTP data fetcher plugins Guzzle like this:

    source:
      plugin: url
      data_fetcher_plugin: http
      data_parser_plugin: json
      urls:
        - 'https://example.com/api/something'
      request_options:
        timeout: 120
    

    I'm setting the issue status to β€œClosed (outdated)”. Please don't hesitate to reopen if I'm wrong / misunderstanding something.

Production build 0.69.0 2024