Method tokenize

codegeex/megatron/tokenizer/tokenizer.py:143–144 · view source on GitHub ↗

(self, text)

Source from the content-addressed store, hash-verified

141
142	@abstractmethod
143	def tokenize(self, text):
144	pass
145
146	def detokenize(self, token_ids):
147	raise NotImplementedError(

generate_samples_input_from_fileFunction · 0.45

generate_samples_evalFunction · 0.45

generate_samples_interactive_code_contestFunction · 0.45

generate_samples_interactiveFunction · 0.45

run_generation_distributedFunction · 0.45

mainFunction · 0.45

no outgoing calls

mainFunction · 0.36