Functions — github.com/Duxiaoman-DI/XuanYuan

hub / github.com/Duxiaoman-DI/XuanYuan / functions

Functions52 in github.com/Duxiaoman-DI/XuanYuan

⨍ Functions 52 ◇ Types & classes 6

↓ 20 callersFunctionlog_dist

定义日志函数，只给特定rank的进程记录日志

llm-code/dxm_llm_main.py:23

↓ 6 callersMethodappend_message

(self, role: str, message: str)

conversation.py:46

↓ 6 callersFunctionformat_example

(df, idx, subject, include_answer=True, cot=False)

FinanceIQ/src/utils.py:15

↓ 4 callersFunctiongen_prompt

(dev_df, subject, prompt_end, num_few_shot=0, tokenizer=None, max_length=2048, cot=False)

FinanceIQ/src/utils.py:36

↓ 4 callersFunctionrun_eval

(model, tokenizer, eval, args)

FinanceIQ/src/utils.py:70

↓ 3 callersFunctionget_conv_template

Get a conversation template.

conversation.py:90

↓ 2 callersMethodget_prompt

Get the prompt for generation.

conversation.py:31

↓ 2 callersFunctionsave_hf_format

保存模型为huggingface格式，以便后续可以用hf.from_pretrained加载

llm-code/dxm_llm_main.py:84

↓ 1 callersMethod__init__

(self, data_path, # 数据集路径 tokenizer, # 分词器实例 max_length,

llm-code/dataset.py:13

↓ 1 callersFunctionafter_train

在整个训练过程结束时执行

llm-code/model_hook.py:146

↓ 1 callersFunctionbefore_train

在训练开始前执行

llm-code/model_hook.py:125

↓ 1 callersFunctioncompute_accuracy

(result_dir='')

FinanceIQ/src/utils.py:174

↓ 1 callersFunctionconcat_multiple_sample_fn

(max_length, pad_token_id)

llm-code/pretrain_data_process.py:46

↓ 1 callersMethodcopy

(self)

conversation.py:52

↓ 1 callersFunctioneval

(subject, dev_df, test_df, num_few_shot, max_length, cot, **kwargs)

FinanceIQ/src/gpt4.py:37

↓ 1 callersFunctionget_access_token

()

FinanceIQ/src/ernie.py:13

↓ 1 callersFunctionget_dataloader_common

用于创建数据加载器（DataLoader）和数据集

llm-code/model_hook.py:55

↓ 1 callersFunctionget_deepspeed_config

(args)

llm-code/config.py:44

↓ 1 callersFunctionget_ds_config

用于获取 DeepSpeed 的配置参数

llm-code/model_hook.py:96

↓ 1 callersFunctionget_ds_model

(args, dataloader_dict)

llm-code/dxm_llm_main.py:35

↓ 1 callersFunctionget_model_common

获取并加载模型文件

llm-code/model_hook.py:32

↓ 1 callersFunctionget_op_lr

获取优化器和学习率

llm-code/model_hook.py:118

↓ 1 callersFunctionget_pt_dataset

用于加载已tokenize后的数据集，用于预训练任务。

llm-code/dataset.py:51

↓ 1 callersFunctionget_response

(inputs)

FinanceIQ/src/chatgpt.py:14

↓ 1 callersFunctionget_response

(inputs)

FinanceIQ/src/ernie.py:18

↓ 1 callersFunctionget_response

(inputs)

FinanceIQ/src/gpt4.py:14

↓ 1 callersFunctionget_tokenizer

加载tokenizer

llm-code/model_hook.py:14

↓ 1 callersFunctioninfer

(model, tokenizer, prompt)

cli_demo.py:24

↓ 1 callersFunctioninfer_vllm

(llm, sampling_params, prompt)

cli_demo.py:16

↓ 1 callersFunctionmain

()

llm-code/dxm_llm_main.py:106

↓ 1 callersFunctionon_epoch_end

在每个训练周期（epoch）结束时执行

llm-code/model_hook.py:139

↓ 1 callersFunctionon_step_end

在每个训练步骤结束时执行

llm-code/model_hook.py:132

↓ 1 callersFunctionparse_args

解析命令行参数

llm-code/model_hook.py:104

↓ 1 callersFunctionparse_arguments

()

llm-code/config.py:5

↓ 1 callersFunctionregister_conv_template

conversation.py:80

↓ 1 callersFunctionseed_all

(seed)

llm-code/dxm_llm_main.py:74

↓ 1 callersFunctiontokenize_and_group_chunk

(data_path, save_dir, tokenizer)

llm-code/pretrain_data_process.py:67

↓ 1 callersFunctiontokenizer_fn

(tokenizer)

llm-code/pretrain_data_process.py:31

↓ 1 callersMethodupdate_last_message

(self, message: str)

conversation.py:49

Method__call__

(self, instances: list)

llm-code/dataset.py:134

Method__getitem__

(self, idx)

llm-code/dataset.py:46

Method__getitem__

(self, idx)

llm-code/dataset.py:90

Method__init__

(self, data_path, # 数据集路径 tokenizer, # 分词器实例 max_length,

llm-code/dataset.py:65

Method__len__

(self)

llm-code/dataset.py:42

Method__len__

(self)

llm-code/dataset.py:86

Functionconcat_multiple_sample

(batch)

llm-code/pretrain_data_process.py:47

(subject, dev_df, test_df, num_few_shot, max_length, cot, **kwargs)

FinanceIQ/src/chatgpt.py:30

Functioneval

(subject, dev_df, test_df, num_few_shot, max_length, cot, **kwargs)

FinanceIQ/src/ernie.py:36

Functioneval

(model, tokenizer, subject, dev_df, test_df, num_few_shot, max_length, cot)

FinanceIQ/src/hf_causal_model.py:13

Functionextract_choice

Always return a choice, even cannot match by regex, to ensure fair comparison to other models.

FinanceIQ/src/utils.py:107

Functiontokenize

(line)

llm-code/pretrain_data_process.py:32