MCPcopy
hub / github.com/Audio-AGI/AudioSep / tokenizer

Function tokenizer

models/CLAP/training/data.py:46–47  ·  view source on GitHub ↗
(text)

Source from the content-addressed store, hash-verified

44
45
46def tokenizer(text):
47 return tokenize(text).squeeze(0)
48
49
50from transformers import RobertaTokenizer

Callers 5

evaluate_clotho_audiocapsFunction · 0.70
preprocessFunction · 0.70
bert_embeddingsFunction · 0.50
Roberta_embeddingsFunction · 0.50
bart_embeddingsFunction · 0.50

Calls 1

tokenizeFunction · 0.90

Tested by

no test coverage detected