Prepare the text before chunking it into parts

Created on 25 April 2024, 7 months ago

Updated 1 June 2024, 6 months ago

Problem/Motivation

When we try to index a rendered item the text can contain a lot of HTML tags at the beginning. That means that the text chunking won't be correct in this case.
So we need to remove tags before chunking

📌 Task

Status

Needs review

Version

1.0

Component

Code

Created by

🇺🇦Ukraine HitchShock Ukraine

Live updates comments and jobs are added and updated live.

Merge Requests

!16Prepare the text before chunking it into parts
Open
🇺🇦Ukraine HitchShock
updated 7 months ago

Comments & Activities

Issue created by @HitchShock
Merge request !16[#3443396] Prepare the text before chunking it into parts → (Open) created by HitchShock
Comment 7 months ago →
🇺🇦Ukraine HitchShock Ukraine
Status changed to Needs review 7 months ago8:45am 25 April 2024
Comment 7 months ago →
🇺🇦Ukraine HitchShock Ukraine
Comment 6 months ago →
🇭🇺Hungary asrob Hungary 🇭🇺 🇪🇺
Hi, I think it's outdated because the new dev version contains this logic and fixes this issue.

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024

Prepare the text before chunking it into parts

Problem/Motivation

Merge Requests

!16Prepare the text before chunking it into partsOpen

Comments & Activities

!16Prepare the text before chunking it into parts
Open