Exception: Failed to insert into collection:

What happens if you set the dimensions to 1024?

I still get the same error

I just changed the Vector Dimension to 1024 and now the error is "TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php)"

the path takes me to the function
/**
* Acts as static factory, and returns a new Response instance.
*
* @param array{object: string, data: array, index: int}>, usage: array{prompt_tokens: int, total_tokens: int}} $attributes
*/
public static function from(array $attributes, MetaInformation $meta): self
{
$embeddings = array_map(fn (array $result): CreateResponseEmbedding => CreateResponseEmbedding::from(
$result
), $attributes['data']);

return new self(
$attributes['object'],
$embeddings,
CreateResponseUsage::from($attributes['usage']),
$meta,
);
}

I just changed the Vector Dimension to 1024 and now i get the error "TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php).

The path takes me to the function
/**
* Acts as static factory, and returns a new Response instance.
*
* @param array{object: string, data: array, index: int}>, usage: array{prompt_tokens: int, total_tokens: int}} $attributes
*/
public static function from(array $attributes, MetaInformation $meta): self
{
$embeddings = array_map(fn (array $result): CreateResponseEmbedding => CreateResponseEmbedding::from(
$result
), $attributes['data']);

return new self(
$attributes['object'],
$embeddings,
CreateResponseUsage::from($attributes['usage']),
$meta,
);
}

I changed the vector Dimension to 1024 and now i get the error "TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php)."

the path takes me to this function
/**
* Acts as static factory, and returns a new Response instance.
*
* @param array{object: string, data: array, index: int}>, usage: array{prompt_tokens: int, total_tokens: int}} $attributes
*/
public static function from(array $attributes, MetaInformation $meta): self
{
$embeddings = array_map(fn (array $result): CreateResponseEmbedding => CreateResponseEmbedding::from(
$result
), $attributes['data']);

return new self(
$attributes['object'],
$embeddings,
CreateResponseUsage::from($attributes['usage']),
$meta,
);
}

I changed the Vertor Dimension to 1024 and now i get the error "TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php)."

the path takes me to this function
/**
* Acts as static factory, and returns a new Response instance.
*
* @param array{object: string, data: array, index: int}>, usage: array{prompt_tokens: int, total_tokens: int}} $attributes
*/
public static function from(array $attributes, MetaInformation $meta): self
{
$embeddings = array_map(fn (array $result): CreateResponseEmbedding => CreateResponseEmbedding::from(
$result
), $attributes['data']);

return new self(
$attributes['object'],
$embeddings,
CreateResponseUsage::from($attributes['usage']),
$meta,
);
}

I changed the Vertor Dimension to 1024 and now i get the error "TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php)."

the path takes me to this function
/**
* Acts as static factory, and returns a new Response instance.
*
* @param array{object: string, data: array, index: int}>, usage: array{prompt_tokens: int, total_tokens: int}} $attributes
*/
public static function from(array $attributes, MetaInformation $meta): self
{
$embeddings = array_map(fn (array $result): CreateResponseEmbedding => CreateResponseEmbedding::from(
$result
), $attributes['data']);

return new self(
$attributes['object'],
$embeddings,
CreateResponseUsage::from($attributes['usage']),
$meta,
);
}

I changed the Vector Dimension to 1024 and now i get the error "TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php)."

The path takes me to this function
/**
* Acts as static factory, and returns a new Response instance.
*
* @param array{object: string, data: array, index: int}>, usage: array{prompt_tokens: int, total_tokens: int}} $attributes
*/
public static function from(array $attributes, MetaInformation $meta): self
{
$embeddings = array_map(fn (array $result): CreateResponseEmbedding => CreateResponseEmbedding::from(
$result
), $attributes['data']);

return new self(
$attributes['object'],
$embeddings,
CreateResponseUsage::from($attributes['usage']),
$meta,
);
}

I changed the Vector Dimension to 1024 and now i get the error "TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php)."

the path takes me to the function
/**
* Acts as static factory, and returns a new Response instance.
*
* @param array{object: string, data: array, index: int}>, usage: array{prompt_tokens: int, total_tokens: int}} $attributes
*/
public static function from(array $attributes, MetaInformation $meta): self
{
$embeddings = array_map(fn (array $result): CreateResponseEmbedding => CreateResponseEmbedding::from(
$result
), $attributes['data']);

return new self(
$attributes['object'],
$embeddings,
CreateResponseUsage::from($attributes['usage']),
$meta,
);
}

I changed the Vector Dimension to 1024 and now i get the error "TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php)."

the path takes me to the function
/**
* Acts as static factory, and returns a new Response instance.
*
* @param array{object: string, data: array, index: int}>, usage: array{prompt_tokens: int, total_tokens: int}} $attributes
*/
public static function from(array $attributes, MetaInformation $meta): self
{
$embeddings = array_map(fn (array $result): CreateResponseEmbedding => CreateResponseEmbedding::from(
$result
), $attributes['data']);

return new self(
$attributes['object'],
$embeddings,
CreateResponseUsage::from($attributes['usage']),
$meta,
);
}

Whatever you're doing to constantly send notifications please stop. It definitely won't make us want to help you.

I am sorry about i dont not see my comment for some weird reason and did not want to leave you with no response

Excessive notifications were sent due to a quirk in the site's spam filtering. User fessouma is not to blame for this.

Okay no problem, I guess some issue with drupal.org then with new users, I see your comments now which oddly weren't showing yesterday. Glad they came through!

So that's good progress then, we have a new issue. It seems the error is the dependency outside the module but probably we are calling it missing some info. Could you enable the stack trace so you can see where in the AI module we are triggering the error please? https://www.drupal.org/docs/develop/development-tools/enable-verbose-err... →

Hi fessouma - judging from your code it seems like the Mistral provider might not be answering correctly. Unfortunatly we are using a not 100% compliant web client for the Mistral client, we have a ticket for it here: https://www.drupal.org/project/ai/issues/3460283 ✨ Build custom client for Mistral or switch to HelgeSverre lib Active .

This means that we can't even catch the error here at all times, so we will not be able to output a nice message or log message what went wrong.

What this means is one of two things could go wrong here:

1. You can not change vector dimensions after they are set. You would need to setup another search database/colleciton then and make sure its 1024.
2. The Mistral API is not responding correctly.

The second point you can easily try by enabling the module "AI API Explorer" and then go to /admin/config/ai/explorers/ai-embeddings

There you can choose your Mistral Embed model and just put anything into the big textarea. If it works you should get 1024 numbers back.

i just changed the error log the first error
TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php)
#0 /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php(45): array_map(Object(Closure), NULL)
#1 /code/vendor/openai-php/client/src/Resources/Embeddings.php(30): OpenAI\Responses\Embeddings\CreateResponse::from(Array, Object(OpenAI\Responses\Meta\MetaInformation))
#2 /code/web/modules/contrib/ai/modules/providers/provider_mistral/src/Plugin/AiProvider/MistralProvider.php(220): OpenAI\Resources\Embeddings->create(Array)
#3 [internal function]: Drupal\provider_mistral\Plugin\AiProvider\MistralProvider->embeddings('# COPYCAT FRIED...', 'mistral-embed', Array)
#4 /code/web/modules/contrib/ai/src/Plugin/ProviderProxy.php(150): ReflectionMethod->invokeArgs(Object(Drupal\provider_mistral\Plugin\AiProvider\MistralProvider), Array)
#5 /code/web/modules/contrib/ai/src/Plugin/ProviderProxy.php(81): Drupal\ai\Plugin\ProviderProxy->wrapperCall(Object(ReflectionMethod), Array)
#6 /code/web/modules/contrib/ai/modules/ai_search/src/Plugin/EmbeddingStrategy/MetadataEmbeddingBase.php(72): Drupal\ai\Plugin\ProviderProxy->__call('embeddings', Array)
#7 /code/web/modules/contrib/ai/modules/ai_search/src/Plugin/EmbeddingStrategy/MetadataEmbeddingBase.php(42): Drupal\ai_search\Plugin\EmbeddingStrategy\MetadataEmbeddingBase->getRawEmbeddings(Array)
#8 /code/web/modules/contrib/ai/modules/ai_search/src/Plugin/search_api/backend/SearchApiAiSearchBackend.php(339): Drupal\ai_search\Plugin\EmbeddingStrategy\MetadataEmbeddingBase->getEmbedding('mistral__mistra...', 'mistral__mistra...', Array, Array, Object(Drupal\search_api\Item\Item))
#9 /code/web/modules/contrib/search_api/src/Entity/Server.php(350): Drupal\ai_search\Plugin\search_api\backend\SearchApiAiSearchBackend->indexItems(Object(Drupal\search_api\Entity\Index), Array)
#10 /code/web/modules/contrib/search_api/src/Entity/Index.php(1006): Drupal\search_api\Entity\Server->indexItems(Object(Drupal\search_api\Entity\Index), Array)
#11 /code/web/modules/contrib/search_api/src/Entity/Index.php(935): Drupal\search_api\Entity\Index->indexSpecificItems(Array)
#12 /code/web/modules/contrib/search_api/search_api.module(117): Drupal\search_api\Entity\Index->indexItems(10)
#13 /code/web/core/lib/Drupal/Core/Cron.php(335): search_api_cron()
#14 /code/web/core/lib/Drupal/Core/Extension/ModuleHandler.php(395): Drupal\Core\Cron->Drupal\Core\{closure}(Object(Closure), 'search_api')
#15 /code/web/core/lib/Drupal/Core/Cron.php(318): Drupal\Core\Extension\ModuleHandler->invokeAllWith('cron', Object(Closure))
#16 /code/web/core/lib/Drupal/Core/Cron.php(159): Drupal\Core\Cron->invokeCronHandlers()
#17 /code/web/core/lib/Drupal/Core/ProxyClass/Cron.php(75): Drupal\Core\Cron->run()
#18 /code/web/core/modules/automated_cron/src/EventSubscriber/AutomatedCron.php(65): Drupal\Core\ProxyClass\Cron->run()
#19 [internal function]: Drupal\automated_cron\EventSubscriber\AutomatedCron->onTerminate(Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#20 /code/web/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php(111): call_user_func(Array, Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#21 /code/vendor/symfony/http-kernel/HttpKernel.php(115): Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch(Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...')
#22 /code/web/core/lib/Drupal/Core/StackMiddleware/StackedHttpKernel.php(66): Symfony\Component\HttpKernel\HttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\LocalRedirectResponse))
#23 /code/web/core/lib/Drupal/Core/DrupalKernel.php(715): Drupal\Core\StackMiddleware\StackedHttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\LocalRedirectResponse))
#24 /code/web/index.php(22): Drupal\Core\DrupalKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\LocalRedirectResponse))
#25 {main}

I also have an error for each content in the content type i want to embed
"Error while trying to render item entity:node/9:en with view mode default for search index AI testing collection: Call to a member function getDimension() on null in Drupal\search_api_ai\Plugin\search_api\data_type\Embeddings->getValue() (line 61 of /code/web/modules/contrib/search_api_ai/src/Plugin/search_api/data_type/Embeddings.php). "

#0 /code/web/modules/contrib/search_api/src/Item/Field.php(568): Drupal\search_api_ai\Plugin\search_api\data_type\Embeddings->getValue('\n\n #1 /code/web/modules/contrib/search_api/src/Plugin/search_api/processor/RenderedItem.php(226): Drupal\search_api\Item\Field->addValue('\n\n #2 /code/web/modules/contrib/search_api/src/Item/Item.php(281): Drupal\search_api\Plugin\search_api\processor\RenderedItem->addFieldValues(Object(Drupal\search_api\Item\Item))
#3 /code/web/modules/contrib/search_api/src/Entity/Index.php(987): Drupal\search_api\Item\Item->getFields()
#4 /code/web/modules/contrib/search_api/src/Entity/Index.php(935): Drupal\search_api\Entity\Index->indexSpecificItems(Array)
#5 /code/web/modules/contrib/search_api/search_api.module(117): Drupal\search_api\Entity\Index->indexItems(10)
#6 /code/web/core/lib/Drupal/Core/Cron.php(335): search_api_cron()
#7 /code/web/core/lib/Drupal/Core/Extension/ModuleHandler.php(395): Drupal\Core\Cron->Drupal\Core\{closure}(Object(Closure), 'search_api')
#8 /code/web/core/lib/Drupal/Core/Cron.php(318): Drupal\Core\Extension\ModuleHandler->invokeAllWith('cron', Object(Closure))
#9 /code/web/core/lib/Drupal/Core/Cron.php(159): Drupal\Core\Cron->invokeCronHandlers()
#10 /code/web/core/lib/Drupal/Core/ProxyClass/Cron.php(75): Drupal\Core\Cron->run()
#11 /code/web/core/modules/automated_cron/src/EventSubscriber/AutomatedCron.php(65): Drupal\Core\ProxyClass\Cron->run()
#12 [internal function]: Drupal\automated_cron\EventSubscriber\AutomatedCron->onTerminate(Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#13 /code/web/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php(111): call_user_func(Array, Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#14 /code/vendor/symfony/http-kernel/HttpKernel.php(115): Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch(Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...')
#15 /code/web/core/lib/Drupal/Core/StackMiddleware/StackedHttpKernel.php(66): Symfony\Component\HttpKernel\HttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\LocalRedirectResponse))
#16 /code/web/core/lib/Drupal/Core/DrupalKernel.php(715): Drupal\Core\StackMiddleware\StackedHttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\LocalRedirectResponse))
#17 /code/web/index.php(22): Drupal\Core\DrupalKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\LocalRedirectResponse))
#18 {main}

I created a whole new Database/collection. in the AI API Explore embedding works but when i try to index the content it still get the error

An error has occurred.
Continue to the error page

An AJAX HTTP error occurred.
HTTP Result Code: 500
Debugging information follows.
Path: /batch?id=81&op=do_nojs&op=do
StatusText: error
ResponseText: The website encountered an unexpected error. Try again later.TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 45 of /code/vendor/openai-php/client/src/Responses/Embeddings/CreateResponse.php). OpenAI\Responses\Embeddings\CreateResponse::from(Array, Object) (Line: 30)
OpenAI\Resources\Embeddings->create(Array) (Line: 220)
Drupal\provider_mistral\Plugin\AiProvider\MistralProvider->embeddings('# DOUBLE DECKER TACOS', 'mistral-embed', Array)
ReflectionMethod->invokeArgs(Object, Array) (Line: 150)
Drupal\ai\Plugin\ProviderProxy->wrapperCall(Object, Array) (Line: 81)
Drupal\ai\Plugin\ProviderProxy->__call('embeddings', Array) (Line: 72)
Drupal\ai_search\Plugin\EmbeddingStrategy\MetadataEmbeddingBase->getRawEmbeddings(Array) (Line: 42)
Drupal\ai_search\Plugin\EmbeddingStrategy\MetadataEmbeddingBase->getEmbedding('mistral__mistral-embed', 'mistral__codestral-mamba-latest', Array, Array, Object) (Line: 339)
Drupal\ai_search\Plugin\search_api\backend\SearchApiAiSearchBackend->indexItems(Object, Array) (Line: 350)
Drupal\search_api\Entity\Server->indexItems(Object, Array) (Line: 1006)
Drupal\search_api\Entity\Index->indexSpecificItems(Array) (Line: 935)
Drupal\search_api\Entity\Index->indexItems(5) (Line: 160)
Drupal\search_api\IndexBatchHelper::process(Object, 5, -1, Array) (Line: 296)
_batch_process() (Line: 138)
_batch_do() (Line: 94)
_batch_page(Object) (Line: 52)
Drupal\system\Controller\BatchController->batchPage(Object)
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 638)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 121)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 181)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 53)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 28)
Drupal\Core\StackMiddleware\ContentLength->handle(Object, 1, 1) (Line: 32)
Drupal\big_pipe\StackMiddleware\ContentLength->handle(Object, 1, 1) (Line: 106)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 85)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 36)
Drupal\Core\StackMiddleware\AjaxPageState->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle(Object, 1, 1) (Line: 741)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

So if you have it working in API Explorer, I just tried it and it works for me, though the default embeddings value is wrong, I will change this - could you follow this to hopefully get it working:

1. When you add the database, click the Set Dimensions Manually
2. Add 1024
3. When you add an index, you have to add at least one field of type embeddings.

Then it should work.

I will push a change for the defualt embeddings value.

the Database
So the backend is AI Search, The vector Databse is Milvus DB,
The tokenizer Chat model is Mistral AI - open-mistral-7b,
the embedding Strategy is Metadata enriched chunk
I left the maximum chunk size blank
the minimum chunk overlap is 100
metadata maximum percentage is 30%
The embedding Engine is Mistral AI | mistal embed and the dimension is 1024

The Index
Datasource Content
i selected one content type for the datasource
the cron batch size is 5

I have 2 field to embed; General: Rendered HTML output with type embedding, Content: Title set as a string type

Ok, bug is confirmed - I'll add on top of the great configuration added by @fessouma that the content need to be larger then the max chunk size for this bug to happen.

It does not happen when a smaller chunk size is set.
It does not happen when for instance openai-3.5 is used as the tokenizer.

If you want it working fessouma, as a support request one of the two options above is the way to go.

I will move this into a bug request instead.

@scott_euser - is this ok to work on, or should I rather wait until all the refactoring is done? I'm guessing the error is in the embeddingsstragy in this case.

Fine to work on now yeah, the changes happening to AI Search are all in here now 📌 Refactor Search API integration to pass more of the work to the VBD Provider Active

Okay I finally managed to try out mistral 7b on my local. I could not manage to reproduce the error though unfortunately. I was able to index content without issue.

As noted in your new issue 💬 Issue embedding the content Active I think watching the recording of the talk I gave last night should help: 🌱 #2 Drupal AI Online Meetup Active
Thanks!

Great glad you managed to get it working!

Automatically closed - issue fixed for 2 weeks with no activity.