Token count calculation

Created on 15 May 2024, 7 months ago

Problem/Motivation

Token count calculation is a common feature which could be included in ai_utilities.

I calculate basic token count of code here: https://git.drupalcode.org/project/aidev/-/blob/1.0.x/src/Source.php?ref...
And also have a feature request to calculate prompt token count here Calculate prompt token count Active
It would be nice to improve prompt count calculation to make it more accurate, because basic calculation averages are different for text/code/etc.
Token calculation could also take into account nuances of different LLMs.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Feature request
Status

Active

Version

1.0

Component

Code

Created by

🇱🇹Lithuania mindaugasd

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024