Convert the provided tokens into text (inverse of llama_tokenize()). Args: vocab: The vocabulary to use for tokenization. tokens: The tokens to convert. n_tokens: The number of tokens. text: The buffer to write the text to. text_len_max: The length of the
(
vocab: llama_vocab_p,
tokens: CtypesArray[llama_token],
n_tokens: Union[ctypes.c_int, int],
text: bytes,
text_len_max: Union[ctypes.c_int, int],
remove_special: Union[ctypes.c_bool, bool],
unparse_special: Union[ctypes.c_bool, bool],
/,
)
| 4073 | ctypes.c_int32, |
| 4074 | ) |
| 4075 | def llama_detokenize( |
| 4076 | vocab: llama_vocab_p, |
| 4077 | tokens: CtypesArray[llama_token], |
| 4078 | n_tokens: Union[ctypes.c_int, int], |
| 4079 | text: bytes, |
| 4080 | text_len_max: Union[ctypes.c_int, int], |
| 4081 | remove_special: Union[ctypes.c_bool, bool], |
| 4082 | unparse_special: Union[ctypes.c_bool, bool], |
| 4083 | /, |
| 4084 | ) -> int: |
| 4085 | """Convert the provided tokens into text (inverse of llama_tokenize()). |
| 4086 | |
| 4087 | Args: |
| 4088 | vocab: The vocabulary to use for tokenization. |
| 4089 | tokens: The tokens to convert. |
| 4090 | n_tokens: The number of tokens. |
| 4091 | text: The buffer to write the text to. |
| 4092 | text_len_max: The length of the buffer. |
| 4093 | remove_special: Allow to remove BOS and EOS tokens if model is configured to do so. |
| 4094 | unparse_special: If true, special tokens are rendered in the output.""" |
| 4095 | ... |
| 4096 | |
| 4097 | |
| 4098 | # // |
nothing calls this directly
no outgoing calls
no test coverage detected
searching dependent graphs…