Add json field parser that just dumps the exact response from Open AI

Created on 9 May 2024, about 2 months ago
Updated 14 May 2024, about 1 month ago

Problem/Motivation

We have complex post processing requirements and seeing the actual result from Open ai and being able to process it programmatically would be very useful.

If you have some tips I'd be happy to work on this.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

✨ Feature request
Status

Needs review

Version

1.0

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States nicxvan

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @nicxvan
  • πŸ‡©πŸ‡ͺGermany Marcus_Johansson

    Hi @nicxvan - Depending on what you need here I could think of some possibilities and I could implement all if they are needed:

    1. Adding event - this is something that I have in the backlog already, to add events where you can create custom eventlisteners to change prompts, change settings etc. during runtime based on dymanical values. This could add an event for raw results that you could use for whatever custom thing you want to implement. So, this is already somehing I will add.
    2. Adding a JSON field rule ( https://www.drupal.org/project/json_field β†’ ) - this would come with two different settings I would imagine - getting the full raw response, or getting the JSON response and saving it or have a query that asks specifically for JSON output that gets saved.
    3. Adding the possibility to add another field as a dump for the full request data in serialized or json format. So a dropdown setting for all long string fields

    Does any of these solve you issues. The first one will be done within the next two weeks, the second one makes more sense to add anyway. The third one is easy to add, but I don't know if it would be so useful?

  • πŸ‡ΊπŸ‡ΈUnited States nicxvan

    Thanks for looking here are my thoughts.

    1. I think long term this is great for additional integration and flexibility.
    2. I think this would solve my current issue since I am trying to post process the changes so I can create multiple entities based on the response.
    3. I would just make the option dump it at the bottom of the long text field rather than specifying a second field, but a second field is acceptable too.

  • πŸ‡©πŸ‡ͺGermany Marcus_Johansson

    @nicxvan The events have actually been added today in the main AI Interpolator module, but I will keep this ticket for the JSON field, since the event for verbose data would still require some changes for each plugin, since they are the ones sending verbose data to the event.

    This should be testable now if you get the DEV version of this module and the DEV version of the main AI Interpolator module and you create any of the three possible JSON field types.

  • Status changed to Needs review about 2 months ago
  • πŸ‡©πŸ‡ͺGermany Marcus_Johansson

    You would need to flush the cache after getting those as well, since its based on annotations.

  • πŸ‡ΊπŸ‡ΈUnited States nicxvan

    Ok I downloaded both dev versions, I don't see any options on json fields yet.

  • πŸ‡©πŸ‡ͺGermany Marcus_Johansson

    Could you check if you see the files src/Plugin/AiInterpolatorFieldRules/OpenAiJson*.php <- there should be 3 off them.

    If you see them could you try flushing the cache, they should really show up on the JSON Field ( https://www.drupal.org/project/json_field β†’ )

  • πŸ‡ΊπŸ‡ΈUnited States nicxvan

    There is something strange going on with the project, I think you have to enable dev releases.

    If you edit the main project page then scroll all of the way down and click on issues, then scroll all of the way down again you should see an option for Version options.

    Are both the tagged releases and development version checked (see screenshot)

    I don't see a dev version on the project page and even though I included dev it is only grabbing the rc1 still.

    There are no json field rules.

    If you check that dev version it should make it available.

  • πŸ‡©πŸ‡ͺGermany Marcus_Johansson

    Could you check now if its works better- the DEV version is visible on the first page now on both projects.

  • πŸ‡ΊπŸ‡ΈUnited States nicxvan

    Ok I was able to get the dev version for both.

    I now see ai interpolator for json AND I see the option to dump the full value.

    When I try to save content I get the following error:

    TypeError: Drupal\ai_interpolator\AiInterpolatorRuleRunner::__construct(): Argument #3 ($eventDispatcher) must be of type Symfony\Component\EventDispatcher\EventDispatcher, Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher given, called in /var/www/html/web/core/lib/Drupal/Component/DependencyInjection/Container.php on line 259 in Drupal\ai_interpolator\AiInterpolatorRuleRunner->__construct() (line 42 of modules/contrib/ai_interpolator/src/AiInterpolatorRuleRunner.php).

  • πŸ‡ΊπŸ‡ΈUnited States nicxvan

    The typehint is: \Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher $eventDispatcher

    But you're passing in just '@event_dispatcher'

    and the use statement is Symfony\Component\EventDispatcher\EventDispatcher

    I think you just need to align those, I'm not sure which one you wanted.

  • πŸ‡©πŸ‡ͺGermany Marcus_Johansson

    Yeah, sorry about that a new update is there, with an actual interface instead of a specific eventdispatcher. Could you try to get the last version and see if it works better. Thanks.

  • πŸ‡ΊπŸ‡ΈUnited States nicxvan

    Ok that worked!

    A couple of notes, none of these are critical, but observations.

    1. When checking store full request the request is escaped:
    "content":"[\n {\n \"organization_name\": \"ABC Corp\",
    I'm not sure I would even change this.
    2. When store full request is checked you don't also get the response, though it's embedded so I don't think that should change either.
    3. The Store full request option is not preserved on subsequent edits. Steps to reproduce below.

    1. Edit json field and enable ai interpolator
    2. Check Store full request under advanced settings
    3. Save
    4. Edit same field
    5. Open advanced settings
    6. Store full request is not checked

  • πŸ‡©πŸ‡ͺGermany Marcus_Johansson

    Thanks for finding that, it was a type in fetching the value.

Production build 0.69.0 2024