- Issue created by @mindaugasd
Token count calculation is a common feature which could be included in ai_utilities.
I calculate basic token count of code here: https://git.drupalcode.org/project/aidev/-/blob/1.0.x/src/Source.php?ref...
And also have a feature request to calculate prompt token count here
✨
Calculate prompt token count
Active
It would be nice to improve prompt count calculation to make it more accurate, because basic calculation averages are different for text/code/etc.
Token calculation could also take into account nuances of different LLMs.
Active
1.0
Code