Functions — github.com/OpenNMT/Tokenizer

CodeHub MCPcopy Create free account

hub / github.com/OpenNMT/Tokenizer / functions

Functions261 in github.com/OpenNMT/Tokenizer

⨍ Functions 261◇ Types & classes 32

↓ 92 callersFunctiontest_tok_and_detok

test/test.cc:122

↓ 64 callersFunctiontest_tok

test/test.cc:33

↓ 41 callersMethodTokenizer

src/Tokenizer.cc:158

↓ 30 callersFunctionget_data

test/test.cc:29

↓ 30 callersMethodtokenize

src/Tokenizer.cc:470

↓ 27 callersMethodappend

src/Tokenizer.cc:545

↓ 20 callersFunctiontest_detok

test/test.cc:78

↓ 14 callersMethodsegment

src/Tokenizer.cc:595

↓ 13 callersMethoddetokenize

src/Tokenizer.cc:212

↓ 8 callersFunctionget_characters_info

src/unicode/Unicode.cc:217

↓ 7 callersFunctionappend

include/onmt/Token.h:46

↓ 7 callersMethodload_vocabulary

src/SubwordEncoder.cc:16

↓ 5 callersFunctionadd_final_token

src/Tokenizer.cc:1027

↓ 5 callersMethodadd_token

src/Vocab.cc:21

↓ 5 callersFunctioncp_to_utf8

src/unicode/Unicode.cc:18

↓ 5 callersFunctionget_bigram

src/BPELearner.cc:95

↓ 5 callersFunctionget_char_type

src/unicode/Unicode.cc:146

↓ 5 callersMethodingest

src/BPELearner.cc:59

↓ 5 callersMethodis_placeholder

↓ 5 callersMethodlearn

src/BPELearner.cc:292

↓ 5 callersMethodresize

src/Vocab.cc:61

↓ 5 callersMethodsafe_append

src/Tokenizer.cc:618

↓ 5 callersMethodserialize_tokens

bindings/python/pyonmttok/Python.cc:176

↓ 5 callersMethodset_vocabulary

↓ 5 callersFunctionstarts_with

src/Utils.cc:14

↓ 4 callersMethodBPELearner

src/BPELearner.cc:27

↓ 4 callersMethodVocab

src/Vocab.cc:13

↓ 4 callersMethodadd_from_text

src/Vocab.cc:43

↓ 4 callersFunctioncharacter_iterator

src/unicode/Unicode.cc:50

↓ 4 callersMethoddeserialize_tokens

bindings/python/pyonmttok/Python.cc:189

↓ 4 callersMethoddetokenize_with_ranges

bindings/python/pyonmttok/Python.cc:201

↓ 4 callersFunctionload_model

src/SentencePiece.cc:13

↓ 4 callersFunctionlog_progress

src/ITokenizer.cc:47

↓ 4 callersFunctionsplit_string

src/Utils.cc:25

↓ 4 callersFunctiontest_tok_alphabet

test/test.cc:96

↓ 4 callersMethodtokenize

bindings/python/pyonmttok/Python.cc:140

↓ 4 callersMethodtokenize_batch

bindings/python/pyonmttok/Python.cc:154

↓ 4 callersFunctionupdate_casing

src/Casing.cc:22

↓ 4 callersFunctionutf8len

src/unicode/Unicode.cc:126

↓ 4 callersMethodvalidate

src/Tokenizer.cc:104

↓ 3 callersMethoddetokenize_file

bindings/python/pyonmttok/Python.cc:255

↓ 3 callersMethodescape_append

src/Tokenizer.cc:635

↓ 3 callersFunctionget_lower

src/unicode/Unicode.cc:207

↓ 3 callersFunctionget_most_frequent

src/BPELearner.cc:280

↓ 3 callersMethodingest

bindings/python/pyonmttok/Python.cc:333

↓ 3 callersFunctionis_placeholder

src/Utils.cc:52

↓ 3 callersFunctionplaceholder_starts_with

src/Casing.cc:189

↓ 3 callersFunctionprune_stats

src/BPELearner.cc:232

↓ 3 callersFunctionsize

include/onmt/Vocab.h:44

↓ 3 callersMethodtokenize_file

bindings/python/pyonmttok/Python.cc:239

↓ 3 callersMethodunicode_length

src/Token.cc:13

↓ 2 callersFunctionToken

include/onmt/Token.h:40

↓ 2 callersFunction_check_subword

bindings/python/test/test.py:446

↓ 2 callersFunctionadd_tokenization_options

cli/tokenization_args.h:7

↓ 2 callersFunctionbuild_tokenization_options

cli/tokenization_args.h:63

↓ 2 callersFunctioncasing_to_char

src/Casing.cc:155

↓ 2 callersFunctionchar_to_casing

src/Casing.cc:172

↓ 2 callersFunctioncheck_dropout

↓ 2 callersMethoddetokenize_stream

src/ITokenizer.cc:211

↓ 2 callersMethodencode_and_annotate

↓ 2 callersFunctionends_with

src/Utils.cc:19

↓ 2 callersMethodflush_feature

src/Tokenizer.cc:644

↓ 2 callersFunctionget_case_type

src/unicode/Unicode.cc:133

↓ 2 callersFunctionget_casing_from_markup

src/Casing.cc:211

↓ 2 callersFunctionhas_connected_uppercase

Returns true if the token at offset can be connected to an uppercase token or capital letter. This useful to avoid closing an uppercase region if inte

src/Casing.cc:246

↓ 2 callersMethodingest_token

bindings/python/pyonmttok/Python.cc:338

↓ 2 callersFunctionint_to_hex

src/Utils.cc:75

↓ 2 callersMethodis_placeholder

src/Tokenizer.cc:1143

↓ 2 callersFunctionlowercase_token

src/Casing.cc:72

↓ 2 callersFunctionnormalize_nfc

test/test.cc:12

↓ 2 callersFunctionread_tokens

src/ITokenizer.cc:225

↓ 2 callersMethodtokenize_stream

src/ITokenizer.cc:178

↓ 2 callersFunctionutf8_to_cp

src/unicode/Unicode.cc:29

↓ 2 callersFunctionwrite_case_markup

src/Casing.cc:221

↓ 2 callersFunctionwrite_tokens

src/ITokenizer.cc:278

↓ 1 callersMethodSentencePieceLearner

src/SentencePieceLearner.cc:8

↓ 1 callersFunction_get_long_description

bindings/python/setup.py:12

↓ 1 callersFunction_get_project_version

bindings/python/setup.py:18

↓ 1 callersFunction_maybe_add_library_root

(lib_name, header_only=False)

bindings/python/setup.py:27

↓ 1 callersMethodadd_from_stream

src/Vocab.cc:54

↓ 1 callersMethodannotate_token

src/Tokenizer.cc:355

↓ 1 callersMethodannotate_tokens

src/Tokenizer.cc:390

↓ 1 callersMethodappend_to_feature

src/Tokenizer.cc:653

↓ 1 callersFunctionbuild_placeholder

src/Casing.cc:216

↓ 1 callersFunctionbuild_sp_tokenizer

bindings/python/pyonmttok/Python.cc:277

↓ 1 callersFunctioncreate_temp_file

bindings/python/pyonmttok/Python.cc:400

↓ 1 callersFunctioncreate_token

bindings/python/pyonmttok/Python.cc:438

↓ 1 callersMethodcurrent_length

src/Tokenizer.cc:580

↓ 1 callersMethoddetokenize

src/ITokenizer.cc:158

↓ 1 callersMethoddetokenize

bindings/python/pyonmttok/Python.cc:228

↓ 1 callersMethodfinalize_tokens

src/Tokenizer.cc:1040

↓ 1 callersFunctionget_case

include/onmt/unicode/Unicode.h:131

↓ 1 callersFunctionget_case_markups

src/Casing.cc:269

↓ 1 callersFunctionget_case_v2

src/unicode/Unicode.cc:202

↓ 1 callersFunctionget_default_id

include/onmt/Vocab.h:74

↓ 1 callersFunctionget_inv_char_frequency

src/BPELearner.cc:257

↓ 1 callersFunctionget_next_main_char

src/Tokenizer.cc:759

↓ 1 callersFunctionget_pair_statistics

src/BPELearner.cc:183

↓ 1 callersFunctionget_random_generator_seed

src/Utils.cc:70

↓ 1 callersFunctionget_script

src/unicode/Unicode.cc:278

next →1–100 of 261, ranked by callers