Function token_count

eval_heldout/rewoo/nodes/LLMNode.py:47–48 · view source on GitHub ↗

(text)

Source from the content-addressed store, hash-verified

45	llama_tokenizer = Tokenizer.from_pretrained('THUDM/agentlm-70b')
46
47	def token_count(text):
48	return len(tokenizer.encode(text))
49
50	def get_prompt(conv: Conversation) -> str:
51	if conv.name == 'openchat':

call_llmMethod · 0.85

encodeMethod · 0.80

no test coverage detected