Design an API for sources to define their own filter sets

Created on 15 July 2024, 5 months ago

Problem/Motivation

Right now, the filters built into Project Browser are hard-coded. Every source is assumed to have categories, and that concepts like "covered by security advisories" and "maintenance status" are always applicable.

With the addition of the recipe source, and the possibility of custom sources (including, say, sources that could pull projects from an organization's private GitLab), I'm not sure we should be assuming that these filters always make sense.

Proposed resolution

Let sources define which filters they will respect, and then leave it entirely up to the frontend to render them.

A source should be able to return a JSON-encodable object with the filter definitions. There are four possible types of filters we could start with:

  • text: A string of text the user could enter, which the source could then use as a partial match against, say, the project's title or description.
  • choice: A list of several choices (an associative array), of which the user can choose only one.
  • multiple_choice: A list of several choices (an associative array) of which the user can choose zero or more.
  • boolean: A filter that is either enabled, or not. For example: only show projects that are covered by security advisories, or show everything.

Imagine something like this being sent to the frontend:

{
    "name": {
        "_type": "text",
        "name": "Name of the project to search for",
        "value": "Some default value"
    },
    "categories": {
        "_type": "multiple_choice",
        "name": "Categories",
        "choices": {
            "ax": "Admin Experience",
            "commerce": "Commerce",
            "seo": "SEO"
        },
        "value": ["commerce", "sEO"]
    },
    "status": {
        "_type": "choice",
        "name": "Maintenance Status",
        "choices": {
            "active": "Actively maintained",
            "minimal": "Minimally maintained"
        },
        "value": "active"
    },
    "security_advisories": {
        "_type": "boolean",
        "name": "Only show projects covered by security advisories",
        "value": true
    }
}

The frontend would be entirely responsible for rendering these and controlling how the user interacts with them. When one of the filters is changed, the frontend would encode the user's choices as an HTTP string like this:

name=Awesome&categories[0]=ax&categories[1]=seo&status=minimal&security_advisories=0

...and send that to the backend, which would then return a new list of projects matching those filters.

This will take extensive refactoring of the Svelte code, to make it capable to rendering these types of filters and ensuring they can be decoded and encoded correctly.

To make things easier, I propose we do NOT allow custom filter types. Only these four, to begin with, or even fewer. We can add more later if we need to. The source should have zero ability to control the appearance or UX of the filters.

Feature request
Status

Active

Version

2.0

Component

Code

Created by

🇺🇸United States phenaproxima Massachusetts

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @phenaproxima
  • 🇺🇸United States phenaproxima Massachusetts
  • 🇺🇸United States drumm NY, US

    This might be a follow-up issue - for the Drupal.org source, this metadata should come from Drupal.org itself, not be hard-coded in the client code.

    For example, we might merge maintenance status & development status, or their options in the future. We should be able to do that, including changing what’s rolled up into more-simplified "show maintained / show everything” filters.

  • Merge request !548Redesign filter api → (Open) created by phenaproxima
  • Pipeline finished with Failed
    5 months ago
    Total: 492s
    #225816
  • Pipeline finished with Canceled
    5 months ago
    #225823
  • Pipeline finished with Failed
    5 months ago
    Total: 399s
    #225820
  • Pipeline finished with Canceled
    5 months ago
    Total: 86s
    #225824
  • Pipeline finished with Failed
    5 months ago
    Total: 1250s
    #225825
  • Pipeline finished with Success
    5 months ago
    Total: 549s
    #225843
  • 🇺🇸United States drumm NY, US

    This should also extend to sort orders. For example, the default sort order being only project usage could be augmented to consider other factors, like minimal maintenance being ranked lower. And other sources won’t have usage information.

  • 🇺🇸United States chrisfromredfin Portland, Maine
  • 🇪🇸Spain fjgarlin

    Re sorting, the base plugin offers the defaults: https://git.drupalcode.org/project/project_browser/-/blob/2.0.x/src/Plug...

    and these can be changed per plugin as needed: https://git.drupalcode.org/project/project_browser/-/blob/2.0.x/src/Plug...

    --

    Adapting what we currently have to what @phenaproxima is suggesting shouldn't be too difficult in the backend. Maybe we can just add a method in the interface that will force all plugins to implement it and return that json info for filters and sorting.

    I see most of the work happening in the front-end. In fact, I'd probably start there once we agree on "the contract", and once the front-end is ready we can adapt all the plugins.

    Also, even tho I mentioned the part of the sorting above, I agree that this should also be moved to be part of "the contract", so filters and sorting are all included.

  • 🇺🇸United States phenaproxima Massachusetts

    I actually think sorting is fine as-is. It's inherently less complicated than filtering; how many different ways do we need to render a sort control? What could it contain except for "this the field to sort on, and this is its human-readable label"? That's already covered...

  • 🇪🇸Spain fjgarlin

    Happy with that too. Sorting is easy as it has defaults and overriding is just providing a relatively simple array.

  • 🇺🇸United States drumm NY, US

    For Drupal.org, we could always sort by “relevancy” which is

    • Project usage, potentially with boosts added later for moving projects up/down based on signals that reliably show module quality
    • When searching for text, the primary ranking is the text search (balanced among title/description/etc) and then usage and any other factors

    Like we do on https://www.drupal.org/project/project_module . I’m not sure we actually need any other sorting options. The point for this issue would be making sure we’re not hard-coding anything like “most installed”

  • 🇺🇸United States phenaproxima Massachusetts

    We're not going to add any kind of filtering or sorting by most installed; this is strictly about defining and implementing the PHP API, and adjusting the JavaScript to match.

  • 🇪🇸Spain fjgarlin

    This is the current map for sorting: https://git.drupalcode.org/project/project_browser/-/blob/2.0.x/src/Plug...

    Left key is project_browser sorting, right value is hiw it’s tranlated into the sort field for the endpoint. Note that the value is not a direct field name but an alias created in the search api index.

  • Pipeline finished with Failed
    5 months ago
    Total: 440s
    #228936
  • 🇮🇳India kunal.sachdev

    kunal.sachdev made their first commit to this issue’s fork.

  • Pipeline finished with Failed
    4 months ago
    Total: 405s
    #233098
  • Pipeline finished with Failed
    4 months ago
    Total: 417s
    #233653
  • Pipeline finished with Failed
    4 months ago
    Total: 527s
    #233816
  • Pipeline finished with Failed
    4 months ago
    Total: 571s
    #234834
  • Pipeline finished with Failed
    4 months ago
    Total: 623s
    #236996
  • Pipeline finished with Failed
    4 months ago
    Total: 638s
    #237004
  • Pipeline finished with Failed
    4 months ago
    Total: 644s
    #237335
  • Pipeline finished with Failed
    4 months ago
    Total: 635s
    #244614
  • 🇦🇺Australia pameeela

    @phenaproxima is this still a Drupal CMS blocker now that we are using a standalone browser?

  • 🇺🇸United States phenaproxima Massachusetts

    I'm not sure this should ever have been a Drupal CMS stable blocker, honestly. It should probably block a beta of Project Browser, but I don't see why it need block Drupal CMS.

  • 🇺🇸United States drumm NY, US

    Since project browser is installed with Drupal CMS. We do need to get the Drupal.org API as final as possible. I fully expect there are more API changes we might want/need to make server side, since there wasn’t a real API design phase. Some of those API changes might be breaking changes. Once Drupal CMS 1.0 is released with project browser, we’re committed to no longer making breaking API changes. Finalizing the API has a good chance of also requiring changes to the project browser plugin API.

Production build 0.71.5 2024