image similarity search (reverse image search) aka search with an image on Drupal

Created on 15 January 2025, 6 months ago

Problem/Motivation

Got this request yesterday:

We would like, when users upload an image that it checks if this image was not already uploaded (in one shape or another) or if a similar image exists, that we might choose that.

Proposed resolution

1. For every image, do a embedding call on the image (not on the metadata)
2. on file upload create a new widget (reduce duplicates input widget, or something similar)
So that when a users uploads an image, we can do the following:
3. Embed the new image,
4. Query for similar images (similarity search).
5. Show a list of similar images (if threshold > X) which the user can then
6. Choose existing image or ignore and use my newly uploaded (image and vector).

Required:
1. IMage embedding model
2. Media embedding index (milvus/pinecone?)
3. widget that triggers the similarity search

EXTRA
This could also give you a search icon like google has for reverse image search.

✨ Feature request
Status

Needs work

Version

1.0

Component

Other Submodules

Created by

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @wouters_f
  • πŸ‡©πŸ‡ͺGermany marcus_johansson

    The idea is awesome and would be great in the media library as well I think!

    Is the idea to have this in a separate module, external contrib module or to be built into one of the modules - for Field Validation it makes, sense, but that is a very limited usage of this.

    I guess you know already - but the image embeddings is already in core and you can use AWS Provider for it. We should do another sweep of providers that might have added this, that we already support.

  • πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

    I think
    - a separate module makes absolute sense I was dreaming of ai_search_image_with_image or ai_image_search)
    - i also think it could be a search field for searching the media library.
    - I have not tested it so I was not aware it could be done in the AWS provider. As I personally don't have a AWS account.

  • Status changed to Active 8 days ago
  • πŸ‡ΊπŸ‡ΈUnited States Kristen Pol Santa Cruz, CA, USA

    Note that "Needs work" is for when there is code that needs to be updated

    ===

    We are doing some issue management housekeeping and adding/removing components.

    We are removing the "Other Submodules" component and want people to categorize issues with the best module/submodule component.

    Moving this issue to "...to be triaged" for prioritization.

    See πŸ“Œ Update AI module project components Active for more details.

  • πŸ‡ΊπŸ‡ΈUnited States Kristen Pol Santa Cruz, CA, USA
  • πŸ‡©πŸ‡ͺGermany a.dmitriiev

    Some features not necessary require AI to be involved, the packages like this https://github.com/jenssegers/imagehash can help to achieve the goal with PHP only without spending tokens and calling LLMs. Just a remark.

  • πŸ‡©πŸ‡ͺGermany marcus_johansson

    Moving to Drupal AI Initiative

Production build 0.71.5 2024