Discussions

Hi there

From the documentation, I understand that voyage uses llama2 tokenizer. Is this also true for the voyage-code-2 model? Also, which specific llama2 tokenizer used - is it meta-llama/Llama-2-70b-hf? Asking since I want to use a tokenizer in RecursiveCharacterTextSplitter.from_huggingface_tokenizer before passing the chunked texts to the API for embedding.

Thanks