File preprocess.py

codegeex/mindspore/src/preprocess.py:None–None · view source on GitHub ↗

Source from the content-addressed store, hash-verified

1	# Copyright 2020 Huawei Technologies Co., Ltd
2	#
3	# Licensed under the Apache License, Version 2.0 (the "License");
4	# you may not use this file except in compliance with the License.

nothing calls this directly

JIEBATokenizerClass · 0.90

tokenize_wikiFunction · 0.85

tokenize_lambadaFunction · 0.85

package_fileFunction · 0.85

from_pretrainedMethod · 0.80

existsMethod · 0.45

no test coverage detected