- Issue created by @yautja_cetanu
- 🇬🇧United Kingdom MrDaleSmith
I've had a go at getting up to speed on this, but I've hit the first stumbling block: the majority of the providers (OpenAPI and non-OpenAI) are using https://github.com/openai-php/client as a PHP client to send and receive requests from the LLM. The Realtime API is connected to through the /realtime endpoint, and this endpoint is not supported by the PHP client.
There is no issue raised on the client for adding support for realtime, so it is unlikely work on this has started yet - this is a new experimental feature in beta so it is unlikely there are alternative libraries that can be used. It looks like we will either need to roll our own solution, or face an extended wait for one to appear.
The other issue is that the realtime api has this warning in its documentation:
Real time audio is heavily affected by network conditions, and reliably delivering real time audio to a server (e.g. from a mobile client to a backend) at scale is a challenge when network conditions are unpredictable.
For this reason, if you are building client-side or telephony applications where you do not have control over the reliability of the network, for production use cases we recommend that you evaluate a purpose-built third-party solution, such as our partners' integrations listed below.
As a Drupal module, we will have zero control over the network conditions it is used under, this may cause us issues. Equally it is unlikely we can request all users register with an additional third-party service, and the audio traffic maybe be minimal given that we will be using it for sporadic interactions rather than protracted conversations. It may be something to bear in mind, however.
- 🇬🇧United Kingdom yautja_cetanu
Ok at the very least then this makes me think if we do this we should do this as a module outside of the AI module