Implement accurate token calculation

Problem/Motivation

Continue on ✨ Calculate prompt token count Active to implement accurate token calculation.

And next ✨ Add maximum prompt length configuration and exception Active

Steps to reproduce

Proposed resolution

Maybe this

• 📌 Remove tokenizer-x, refactor token chunking and tokenizer into services, and use those Fixed
• 📌 Change rajentrivedi/tokenizer-x dependency back to release instead of specific commit once new release is made Postponed

as in ai → module.

Remaining tasks

User interface changes

API changes

Data model changes