Remove tokenizer-x, refactor token chunking and tokenizer into services, and use those

Merge request https://git.drupalcode.org/project/ai/-/merge_requests/26

✨ Implement accurate token calculation Active

👀

Added the ability to get a list of supported models from the tokenizer service (as it intersects with the available models in the AI providers list)

Merged into ai-search branch as discussed in slack (after refactor into base ai core module)

Automatically closed - issue fixed for 2 weeks with no activity.