MCPcopy
hub / github.com/e-p-armstrong/augmentoolkit / count_tokens

Function count_tokens

generation/core_components/chunking.py:206–207  ·  view source on GitHub ↗
(message)

Source from the content-addressed store, hash-verified

204
205
206def count_tokens(message):
207 return len(tokenizer.encode(message))
208
209
210def count_tokens_specific_model(model):

Callers 11

check_token_lengthFunction · 0.90
innerFunction · 0.90
find_message_exceeding_thresholdFunction · 0.90
simple_chat_loopFunction · 0.70
chunking_algorithm_fileFunction · 0.70
chunking_algorithm_strFunction · 0.70
format_conversation_historyFunction · 0.70
submit_chat_to_serverFunction · 0.50
submit_chat_to_serverFunction · 0.50
submit_chat_to_server_streamingFunction · 0.50
rptoolkit_pipelineFunction · 0.50

Calls

no outgoing calls

Tested by

no test coverage detected