Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/THUDM/GLM
/ tokenization.py
File
tokenization.py
data_utils/tokenization.py:None–None ·
view source on GitHub ↗
Source
from the content-addressed store, hash-verified
1
# coding=utf-8
2
# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
3
#
4
# Licensed under the Apache License, Version 2.0 (the "License");
Callers
nothing calls this directly
Calls
2
prep_command_tokens
Function · 0.85
prep_type_tokens
Function · 0.85
Tested by
no test coverage detected