hub / github.com/biuping/ChatBot / preprocess_raw_data

Function preprocess_raw_data

gpt_2/gpt_train.py:107–140 · view source on GitHub ↗

对每一个轮次对话进行处理, [CLS]表示dialog开始 [SEP]表示这轮对话中的句子分隔形式："[CLS]utterance1[SEP]utterance2[SEP]utterance3[SEP]" :param hparams: 超参数 :param tokenizer: :param n_ctx: GPT-2上下文窗口大小

(hparams, tokenizer, n_ctx)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 1

mainFunction · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected