Functions — github.com/DLLXW/baby-llama2-chinese

CodeHub MCPcopy Create free account

hub / github.com/DLLXW/baby-llama2-chinese / functions

Functions108 in github.com/DLLXW/baby-llama2-chinese

⨍ Functions 108◇ Types & classes 14

↓ 22 callersMethodencode

(self, s: str, bos: bool = False, eos: bool = False)

chatglm_tokenizer/tokenization_chatglm.py:34

↓ 13 callersMethodinfo

(self, message: str, std_out: bool=True, save_to_file: bool=False)

data_clean/logger.py:74

↓ 11 callersFunctionwrite_single_parquet_file

将dataframe写到单独的parquet file中

data_clean/clear.py:85

↓ 6 callersFunctiondelete_file

询问删除文件

data_clean/clear.py:30

↓ 5 callersMethodget_command

chatglm_tokenizer/tokenization_chatglm.py:80

↓ 4 callersMethod__init__

(self, dim: int, eps: float)

↓ 2 callersMethodconfigure_optimizers

(self, weight_decay, learning_rate, betas, device_type)

↓ 2 callersMethoddecode

(self, t: List[int])

chatglm_tokenizer/tokenization_chatglm.py:43

↓ 2 callersFunctionextract_Ngram

获取一个句子的n_grama return： ngram_counter： key = ('w1 w2 ... wn', n_gram), value: count of key

data_clean/functions.py:372

↓ 2 callersMethodforward

↓ 2 callersMethodgenerate

Take a conditioning sequence of indices idx (LongTensor of shape (b,t)) and complete the sequence max_new_tokens times, feeding the p

↓ 2 callersFunctionget_bleu4_score

获取bleu4分数

data_clean/functions.py:323

↓ 2 callersMethodget_features

文本预处理(正则、分词、滑窗) 滑窗的目的是增加上下文。预处理视情况进行增删 s: 文档 return: List[str] 分词后的文档

data_clean/functions.py:126

↓ 2 callersFunctionremove_duplicate_punctuation

删除句子中重复的标点符号、重复的空格，同时将换行变为特殊字符'\n'

data_clean/clear.py:43

↓ 2 callersFunctionrepair_song_album_list

data_clean/functions.py:182

↓ 2 callersFunctionrepeat_kv

torch.repeat_interleave(x, dim=2, repeats=n_rep)

↓ 2 callersFunctionreshape_for_broadcast

(freqs_cis: torch.Tensor, x: torch.Tensor)

↓ 2 callersMethodvocab_size

chatglm_tokenizer/tokenization_chatglm.py:107

↓ 1 callersMethod__init__

(self, model_path: str)

chatglm_tokenizer/tokenization_chatglm.py:11

↓ 1 callersMethod_convert_id_to_token

Converts an index (integer) in a token (str) using the vocab.

chatglm_tokenizer/tokenization_chatglm.py:123

↓ 1 callersFunction_get_doc_mini_hash

获取一段文本的mini hash

data_clean/functions.py:51

↓ 1 callersMethod_norm

↓ 1 callersMethodadd_doc

添加文档， index：文档的索引 doc: 文档本身

data_clean/functions.py:78

↓ 1 callersMethodadd_doc

添加文档， index：文档的索引 doc: 文档本身

data_clean/functions.py:137

↓ 1 callersFunctionapply_rotary_emb

( xq: torch.Tensor, xk: torch.Tensor, freqs_cos: torch.Tensor, freqs_sin: torch.Tensor )

↓ 1 callersMethodconvert_id_to_token

Converts an index (integer) in a token (str) using the vocab.

chatglm_tokenizer/tokenization_chatglm.py:56

↓ 1 callersMethodconvert_token_to_id

Converts a token (str) in an id using the vocab.

chatglm_tokenizer/tokenization_chatglm.py:50

↓ 1 callersMethoddecode_tokens

(self, tokens: List[str])

chatglm_tokenizer/tokenization_chatglm.py:46

↓ 1 callersMethodget_duplicate_indexs

返回所有的重复文档索引

data_clean/functions.py:100

↓ 1 callersFunctionget_logger

(filename, verbosity=1, name=None)

↓ 1 callersFunctionget_logger

(filename, verbosity=1, name=None)

↓ 1 callersFunctionget_lr

↓ 1 callersFunctionget_lr

↓ 1 callersFunctionget_path_of_suffix_files

获取指定目录下下指定后缀的所有文件的绝对路径

data_clean/functions.py:307

↓ 1 callersMethodget_prefix_tokens

chatglm_tokenizer/tokenization_chatglm.py:158

↓ 1 callersFunctioninit_model

↓ 1 callersFunctioninit_model

pretrain.py:129

↓ 1 callersFunctionprecompute_freqs_cis

(dim: int, end: int, theta: float = 10000.0)

↓ 1 callersFunctionprocess_baike

(response_less_word: int=15)

data_clean/clear.py:419

↓ 1 callersFunctionread_and_write_template_baike

处理数据读写模板，需要提供一个回调函数call_back， read_file: 原始数据文件 write_to_file：处理后的要保存数据文件 call_back：函数输入一个字符串，输出一个处理后的字典dict，如果输入的字符串为无效数据，请返回None

data_clean/clear.py:362

↓ 1 callersFunctionread_and_write_template_wiki

处理数据读写模板，需要提供一个回调函数call_back， read_file: 原始数据文件 write_to_file：处理后的要保存数据文件 call_back：函数输入一个字符串，输出一个处理后的字典dict，如果输入的字符串为无效数据，请返回None

data_clean/clear.py:266

↓ 1 callersFunctionremove_dataset_duplicate_rows

使用min_hash删除数据集中重复的部分

data_clean/clear.py:145

↓ 1 callersFunctionrepair_song_album

修复一条文本的'歌曲'和'专辑'的spo。对于歌曲x（subject）的关系歌手、作词、作曲，x必须同时存在于song和album中

data_clean/functions.py:165

↓ 1 callersFunctionsft_process

sft_data_process.py:7

↓ 1 callersMethodtokenize

chatglm_tokenizer/tokenization_chatglm.py:31

↓ 1 callersFunctiontrain_epoch

↓ 1 callersFunctiontrain_epoch

Method__getitem__

(self, index: int)

dataset_sft.py:27

Method__getitem__

(self, index: int)

(self,df,tokenizer ,max_length=256 ,prompt_max_len=128 ,ans

dataset_sft.py:10

(self,data_path_lst,max_length=256,memmap=False)

(self, args: ModelArgs)

(self, dim: int, hidden_dim: int, multiple_of: int, dropout: float)

(self, layer_id: int, args: ModelArgs)

(self, params: ModelArgs)

(self, vocab_file, padding_side="left", clean_up_tokenization_spaces=False, **kwargs)

chatglm_tokenizer/tokenization_chatglm.py:68

(self, logger_name: str, level=logging.DEBUG, std_out: bool=True, save2file: bool=False, file_name: str=None)

data_clean/logger.py:9

获取一个数据集中所有重复（相似的超过threshold）的index，输入为：list[str]，一个str元素为一段文本(doc) 如输入： [a, b, c, d, c, d, e] 返回：{4, 5} (后面两个 c, d 的index) M

data_clean/functions.py:62

threshold: 汉明距离阈值 f: 哈希值的长度

data_clean/functions.py:114

dataset_sft.py:25

Method_convert_token_to_id

Converts a token (str) in an id using the vocab.

chatglm_tokenizer/tokenization_chatglm.py:119

Method_init_weights

Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs:

chatglm_tokenizer/tokenization_chatglm.py:196

Method_tokenize

(self, text, **kwargs)

chatglm_tokenizer/tokenization_chatglm.py:116

Methodbuild_inputs_with_special_tokens

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

chatglm_tokenizer/tokenization_chatglm.py:171

Methodbuild_prompt

(self, query, history=None)

chatglm_tokenizer/tokenization_chatglm.py:162

Functionconvert_en_punctuation_to_zh_punct

将句子中的英文标点替换文中文标点

data_clean/clear.py:62

Methodconvert_tokens_to_string

(self, tokens: List[str])

chatglm_tokenizer/tokenization_chatglm.py:127

(self, message: str, std_out: bool=True, save_to_file: bool=False)

data_clean/logger.py:80

Methodeos_token

chatglm_tokenizer/tokenization_chatglm.py:99

Methodeos_token_id

chatglm_tokenizer/tokenization_chatglm.py:103

(self, message: str, std_out: bool=True, save_to_file: bool=False)

data_clean/logger.py:92

Methodestimate_mfu

estimate model flops utilization (MFU) in units of A100 bfloat16 peak FLOPS

export the model weights in fp32 into .bin file to be read from C

Functionf1_p_r_compute

spo_list: [ [(s,p,o)...], [(s,p,o)]], 每一行[(s,p,o)...]为一个句子中的spo 计算spo的f1分数，精确率，召回率，

data_clean/functions.py:158

Functionfixed_response

修复被截断的回答，从末尾往回找第一个结束标点

data_clean/functions.py:228

Functionfixed_space

单个空格删除，连续两个空格保留一个

data_clean/functions.py:243

( self, x: torch.Tensor, freqs_cos: torch.Tensor, freqs_sin: torch.Tensor,

(self, x, freqs_cos, freqs_sin)

(self, tokens: torch.Tensor, targets: Optional[torch.Tensor] = None)

Functionget_free_space_of_disk

获取指定目录所在磁盘大小，返回单位: GB

data_clean/functions.py:260

Functionget_sentences_dice_similarity

获取两个句子的Dice相似度（Dice similarity） s(a, b) = 2 * len( set(a) & set(b) ) / (len(set(a)) + len(set(b)))

data_clean/clear.py:71

Methodget_vocab

Returns vocab as a dict

chatglm_tokenizer/tokenization_chatglm.py:110

Functionjson_to_dataclass

将json配置文件转换为dataclass >>> example: >>> data_class = json_to_dataclass('my_config.json', 'Config') >>> my_config = data_class() >>

data_clean/functions.py:284

Functionmerge_dataset_as_single_file

将多个数据集合并为一个数据集

data_clean/clear.py:96

Functionmy_average

自定义均值计算，空数组返回0.0

data_clean/functions.py:275

Methodon_epoch_end

在 on_epoch_end 时保存一次模型。 TrainingArguments的 save_strategy 中 epoch 和 steps 不兼容。要实现每隔 save_steps 步保存一次检查点，考虑到磁盘空间大小，最多只保存最近N个检查点。

data_clean/functions.py:38

在打印 n 次日志后清除cuda缓存，适合低显存设备，能防止OOM

data_clean/functions.py:30

Methodpad_token

chatglm_tokenizer/tokenization_chatglm.py:91

Methodpad_token_id

chatglm_tokenizer/tokenization_chatglm.py:95

Functionprocess_baidu

data_process.py:123

Functionprocess_c4

data_process.py:166

Functionprocess_function

data_clean/clear.py:344

Functionprocess_medical

(data_path,name)

data_process.py:22

Functionprocess_wiki

(response_less_word: int=15)

data_clean/clear.py:336

Functionprocess_wiki_clean

data_process.py:8

Functionprocess_wudao

data_process.py:189

Functionremove_dataset_duplicate_rows_simhash

使用sim_hash删除数据集中重复的部分

data_clean/clear.py:198

next →1–100 of 108, ranked by callers