set seeds for reproducibility

gets the list of labels for this data set.

shape: sequence_input_lst: [batch_size, seq_len, num_labels] wordpiece_mask: [batch_size, seq_len, ]

reuse the code block from https://github.com/jiesutd/NCRFpp/blob/105a53a321eca9c1280037c473967858e01aaa43/utils/metric.py#L73 Many tha

get train/dev/test dataloader

get training dataloader

Desc: load data in conll format Returns: [([word1, word2, word3, word4], [label1, label2, label3, label4]), ([word5,

decode inputs to tags Args: char_label_list: list of tuple (word, bmes-tag) Returns: tags Examples: >>> x = [

reuse the code block from https://github.com/jiesutd/NCRFpp/blob/105a53a321eca9c1280037c473967858e01aaa43/utils/metric.py#L73 Many tha

return basic arg parser

Returns tokenized answer spans that better match the annotated answer.

Convert GENIA data to MRC format

Convert MSRA raw data to MRC format

mrc_data_example: { "context": "The chemiluminescent ( CL ) response of interferon - gamma - treated U937 ( IFN - U937 ) cells to sen

Extract flat-ner spans from start/end/match logits Args: start_pred: [seq_len], 1/True for start, 0/False for non-start end_p

main

main

Compute span f1 according to query-based model output Args: start_preds: [bsz, seq_len] end_preds: [bsz, seq_len] mat

test dataset

other dataset func

Args: item: int, idx Returns: tokens: tokens of query + context, [seq_len] token_type_ids: token

Initialize a model, tokenizer and config.

Initialize a model, tokenizer and config.

pad to maximum length of this batch Args: batch: a batch of samples, each contains a list of field data(Tensor): tokens,

Prepare optimizer and schedule (linear warmup and decay)

Prepare optimizer and schedule (linear warmup and decay)

Args: input_ids: bert input tokens, tensor of shape [seq_len] token_type_ids: 0 for query, 1 for context, tensor of s

remove overlapped spans greedily for flat-ner Args: spans: list of tuple (start, end), which means [start, end] is a ner-span Ret

pad to maximum length of this batch Args: batch: a batch of samples, each contains a list of field data(Tensor): tokens,