Set requires_grad=False for each of model.parameters()

list(map(f, x))

:param gold_list: :param pred_list: :param text_list: :param raw_list: :return: dict:

Remove columns that are populated exclusively by pad_token_id

add space between special token :param text: :return:

This was added to get rougeLsum scores matching published rougeL scores for BART and PEGASUS.

src: source text traget: ((ET)) 只对ET进行生成, ((ET)(ET)...) :param tokens: US President George W. Bush told Canadia

Only used by LegacyDataset

Count Bracket Number, 0 indicate num_left = num_right :param tree_str: :return:

:param the_list: [1, 2, 3, 4, 5, 6, 1, 2, 4, 5] :param to_match: [1, 2] :return: [(0, 1), (6, 7)]

Go through the text data by order of src length with a bit of randomness. From fastai repo.

Uses sacrebleu's corpus_bleu implementation.

Calculate rouge using rouge_scorer package. Args: pred_lns: list of summaries generated by model tgt_lns: list of groundtruth summ

:param the_list: [1, 2, 3, 4, 5, 6, 1, 2, 4, 5] :param to_match: [1, 2] :return: [(0, 1), (6, 7)]

pickle.load(path)

Generate Text Span :param generated: :param prefix_tree: :param src_sentence: :param end_sequence_search_toke

Call tokenizer on src and tgt_lines