In Search API we did a very basic approach to chunking where we looked at the advice at the time and just went with that.
Things to think about:
- Can we find Chunking libraries elsewhere so we don't do it in Drupal.
- Do you have whole "Chunking methods" which includes everything all in one plugin
- Or do we make things configurable when you set up the index (Such as 5000 characters vs 500, try and find the end of the sentence, overlap, Do you strip HTML)
- If its configurable should we have sensible defaults so someone doesn't have to understand how this all works.
- What about adding meta data into the chunk (Such as Node Title, Author, Genre, etc)
Fixed
1.0
AI Search
The initial dev version was pushed to 'ai_search' branch.
Automatically closed - issue fixed for 2 weeks with no activity.