File pre_tokenize_glm4.py

train/pre_tokenize_glm4.py:None–None · view source on GitHub ↗

Source from the content-addressed store, hash-verified

1	from transformers import AutoTokenizer, AutoModel, LlamaTokenizer
2	import copy
3	import torch
4	import json, os, random

nothing calls this directly

get_commandMethod · 0.80

mainFunction · 0.70

parse_argsFunction · 0.70

no test coverage detected