Flatten dialogue state from dict to sequence. Args: ds_dict (dict): The dialogue state dict. add_nm (bool, optional): If True, ad

Run model main forward. Args: inputs: A dict mapping keys to corresponding input data. Returns: outputs: A d

Generation from sequence to sequence.

Call Q-TOD model generation.

Call Q-TOD model generation.

Return a nicely extracted title and chosen sentence. :return: pair (title, sentence)

Get the delta of dialogue state from previous turn to current turn. Args: prev_ds_dict (dict): The previous turn dialogue state.

Setup logger. Args: name (str): Logging name. level (int, optional): Logging level. Defaults to 10. Returns: logging

Run inference. Args: model: A generation model. Need to implement `_generation_network` and `_calc_logits`.

Load persistables or parameters. Args: model_path: The path of initial model. is_checkpoint: If true, load parameters

Loads a vocabulary file into a dictionary.

Merge metrics and statistics.

Lower text and remove punctuation, articles and extra whitespace.

Lower text and remove punctuation, articles and extra whitespace.

Normalize retrieve scores Returns: list: Retrieval scores after normalization

Normalize slot values.

Post-process the decoded sequence. Truncate from the first <eos> and remove the <bos> and <eos> tokens. Convert token_ids to words(merge=True

Preprocess utterance or table value. Returns: str: A processed text.

Calculation for recall at k.

Compute the knowledge selection accuracy of query Args: data (BaseDataset): an instance of dataset containing knowledge selection results

Remove substring.

Repeate numpy array or LoD tensor.

Get the last position of elem in lst.

Convert data to LoDTensor.

Merge multiply chunks into output embeddings. Transpose and then reshape the last two dimensions of input tensor x into one dimension

Add linear projection to queries, keys, and values.

Scaled Dot-Product Attention

Construct a batch reader from a record reader.

Build programs. Build training program, evaluation program and inference program. Only use in static graph mode.

Get the logits of BoW task. The network may share weight with token embeddings. Args: enc_out: the output embeddings of

Get the logits of latent recognition task. The network may share weight with latent embeddings. Args: enc_out: the output

Run intervention case by case. Args: ds_dict (dict): The dialogue state dict. ds_delta_dict (dict): The delta of dial

Compute Entity-F1.

Compute Entity-F1.

Convert an example to a record which can be used as the model's input.

Database guidance. Args: ds_dict (dict): The dialogue state. db_match_score (dict): Matching score of predicted value

Return the max F1 score between the guess and *any* answer.

Return the max F1 score between the guess and *any* answer.

Filter out `name` slot values that do not appear in the dialogue history. Args: ds_dict (dict): The dialogue state dict.

Run forward for dense embedding.

Run forward for KAG training.

Yield batches from record reader.

Get best match idx.

Get the feed list of the model for dense embedding.

Get the feed list of the model for KAG training.

Get field values for KAG generation.

Get field values for KAG training.

Get the optimizer of model. Args: args: arguments. lr_scheduler: learning rate scheduler Returns:

Yield sorted batches from record reader.

Get statistics for KAG training.

Get idx for the target.

Get idx for the topk gathering.

Initialize distributed strategy.

Checks whether `chars` is a control character.

Checks whether `chars` is a whitespace character.

Lower slot values for candidates. Args: candidates (list): Lower case candidates.

Add mask for batch_tokens, return out, mask_label, mask_pos; Note: mask_pos responding the batch_tokens after padded;

Mix random negative samples into dataset.

Mix random negative samples into dataset.

Normalize the generated slot values according to the results of querying the database using multiple slots. Args: dom (str): The

Normalize the generated slot values according to the database. Args: ds_dict (dict): The dialogue state dict. utt_lis

Normalize the generated time string. Args: t (str): The time string. Returns: str: The time string after nor

Pad the instances to a given length in batch.

Pad the instances to a given length in batch.

Padding a batch of records and construct model's inputs. This function can be override by its subclass if necessary.

Pad batch records and mask for KAG training.

Parse and return corresponding fields for KAG training.

Parse knowledge list and return corresponding fields.

Parse topic sequence and return corresponding fields.

Post-process generation output. Calculate repetion, reranking. Args: predictions: the generation outputs of the model.

Post-process inference output.

Post-process scoring output. The score is calculated by perplexity (PPL).

Compute precision, recall and f1 given a set of gold and prediction items. :param pred_items: iterable of predicted values :param gold_it

Compute precision, recall and f1 given a set of gold and prediction items. :param pred_items: iterable of predicted values :param gold_it

Read multiply files and yield records.

Read a file which contains numerical data and yield records.

Read a tab separated value file and yield records.

Convert number to english in system utterance. Args: utt (str): The system utterance. Returns: str: The syst

Convert number to english in user utterance. Args: utt (str): The user utterance. Returns: str: The user utt

Set checkpoints for recompute. Args: checkpoints: A list of Variables which need to set as checkpoints.

Shuffle examples.

Mark entity spans in the input.

Mark entity spans in the input.

Add cmdline arguments.