elasticsearch connector unable to connect to cluster due to proxy dependency

Created on 14 November 2022, almost 3 years ago
Updated 8 October 2024, 10 months ago

Problem/Motivation

The environment has proxy that routes all traffic towards forward proxy and further to the internet.
However, this time I do not want to route the traffic outside the network as Drupal and ES servers are hosted in the same network.

Drupal elasticsearch connector is unable to reach the elasticsearch hosted on another VM due to proxy settings.
Drupal and Elasticsearch both are hosted on different servers but in the same subnet.

From the VM console I can run the "noproxy" with curl and I can see cluster info however if i do not mention noproxy, then curl is not working.

How can I tell elasticsearch connector to connect to the elasticsearch using no_proxy values?

Steps to reproduce

bash-5.1# curl --noproxy '*' http://es-search.dev.org.local:9200/
{
"name" : "777",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "eXt4-4ssssssss1k2AE73w",
"version" : {
"number" : "7.17.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "bee8632sdfsdfsdfdsfsfsfsdsddefd4d9ec56bd",
"build_date" : "2022-01-28T08:36:04.875279988Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

💬 Support request
Status

Active

Component

Elasticsearch Connector

Created by

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇳🇿New Zealand murrow

    In the current build (8.0.0-alpha2), I see no implementation that uses Drupal's HTTP proxy settings. This version of Elasticsearch Connector uses elasticsearch/elasticsearch 8, which should support proxies through the ClientBuilder's setHttpClientOptions() method, as described here: https://www.elastic.co/guide/en/elasticsearch/client/php-api/8.15/http-client.html.

    Is it correct to say that proxy support is missing from this module?

  • 🇳🇿New Zealand murrow

    Changing the version number

  • 🇫🇮Finland sokru

    @murrow, you're correct there's currently no support for proxy connection.

  • 🇳🇿New Zealand murrow

    @sokru, I have updated the issue summary with a proposed solution. The dependency injection part will disturb the code a bit, but I don't think the other tasks will have much impact. What are your thoughts?

  • 🇳🇿New Zealand murrow

    Further update to the summary problem section

  • 🇳🇿New Zealand murrow

    Adding test task

  • First commit to issue fork.
  • hexaki changed the visibility of the branch 3321300-allow-defining-proxy to hidden.

  • hexaki changed the visibility of the branch 3321300-allow-defining-proxy to hidden.

  • hexaki changed the visibility of the branch 3321300-allow-defining-proxy to active.

  • Pipeline finished with Success
    about 1 month ago
    Total: 483s
    #542111
  • Pipeline finished with Success
    about 1 month ago
    Total: 524s
    #542791
  • I pushed a solution using ClientBuilder::setHttpClient.
    I also made it possible for Connector extending the StandardConnector to alter the Client without having to redefine it completely.

  • First commit to issue fork.
  • 🇨🇦Canada mparker17 UTC-4

    Sorry for the noise; I haven't had a chance to review the change yet.

    I rebased onto the latest 8.0.x, so the phpstan pipeline wouldn't warn us about a deprecation

  • 🇨🇦Canada mparker17 UTC-4

    @murrow, @hexaki, thank you both very much for your contributions!

    I've had a chance to look at the code in the merge request, and I have some feedback.

    Before I give my feedback, note that I am assuming that, because you moved the Issue Status to "Needs review", you have finished implementing the full solution that you want. If that's not the case (i.e.: you still have work to do), please move it back to "Needs work" until you're done! Thank you! (it's hard for me to tell because I don't have the context that you do — I've never needed to connect to Elasticsearch through a proxy!)

    Without further ado, here is my feedback...

    1. The Issue Title says "Allow defining proxy settings for connection", so I was expecting to see some configuration and/or UI in the Merge Request to do that.
      (note that I may have misunderstood the Issue Title, and my expectation may be wrong!)
      1. The code in the Merge Request seems to change the specifics of how the Elasticsearch Client object gets built. Has the Issue Title become out-of-sync with the solution? If so, can you update the Issue Title?
        • For example, If you're using some other module (e.g.: cURL HTTP Request ) or service (e.g.: Guzzle) to change the proxy settings, please explain what you're doing, so that I understand! (this will also document a solution for other community members who need to do the same thing that you are doing but don't know how!)
    2. In the Issue Summary, under Proposed resolution, it says "We should be able to use the elasticsearch/elasticsearch ClientBuilder::setHttpClientOptions method to add any Drupal HTTP proxy settings into the call if we have Guzzle and Curl as our transports." so I was expecting to see a call to setHttpClientOptions() in the merge request, but I don't see anything like that.
      (note that I may have misunderstood the issue summary and my expectation may be wrong)
      1. If this is no longer the solution we're going for, can you update the issue summary to explain what the new solution is?
    3. Would you be able to update the Remaining tasks in the Issue Summary to explain what else needs to be done? @sokru added the Needs issue summary update tag, please remove when you're done updating the issue summary.
    4. The merge request doesn't have any automated tests. Automated tests ultimately benefit you, because they ensure that future changes to the module contributed by other people won't break the functionality that your site depends on. If you need help writing tests, please ask us! I've added the Needs tests tag, please remove it when you've added some tests.

    Since I have some questions for you, I'm changing the status to Postponed (maintainer needs more info). When you've answered the questions, please move the issue back to Needs review

    Some other notes...

    I fixed a spelling mistake in a docblock in the code.

    I've updated the Issue Summary a bit based on my understanding of the problem and your solution. Please review it, and fix it if I've made a mistake.

    I removed the proxy and proxy variables issue tags, as those are non-standard. See this list of standard tags .

    I changed this to a "Feature request" to reflect that you're asking for a feature that doesn't currently exist in the module. See also @sokru's comment in #4.

Production build 0.71.5 2024