Constructs a BERT Example. Args: sources: List of source texts. target: Target text or None when building an example during inference

Return a list of the tensor's shape, and ensure no None values in list.

Returns a value close to infinity, but is still finite in `dtype`. This is useful to get a very large value that is still zero when multiplied by

Initialize layers to build Transformer model. Args: params: hyperparameter object defining layer sizes, dropout values, etc. train: b

Compute Fbeta score. Args: true_positives: Number of true positive ngrams. selected: Number of selected ngrams. relevant: Number of rel

Get a Counter with the ngrams of the given ID list. Args: ids: np.array or a list corresponding to a single sentence n: n-gram size Retu

Computes tags needed for converting the source into the target. Args: task: tagging.EditingTask that specifies the input.

Returns labels/tags for the original tokens, not for wordpieces.

Computes logits by running x through a linear layer. Args: x: A float32 tensor with shape [batch_size, length, hidden_size] Returns:

Split x into different heads, and transpose the resulting value. The tensor is transposed to insure the inner dimensions hold the correct val

Computes tags when the order of sources is fixed. Args: source_tokens: List of source tokens. target_tokens: List of toke

Tiles a given tensor by beam_size. Args: tensor: tensor to tile [batch_size, ...] beam_size: How much to tile the tensor by. Returns:

Finds the start index of a span of deleted tokens. If `source_token_idx` is preceded by a span of deleted tokens, finds the start ind

Gather beams from nested structure of tensors. Each tensor in nested represents a batch of beams, where beam refers to a single search state (bea

Gather top beams from nested structure.

Computes the phrases that need to be added to the source to get the target. 计算需要被加入进去的短句，且尽可能长 英文是分成word sep=' '，中文是分成 字 sep=''

Return length normalization factor.

Returns truncated version of x according to the self._max_seq_length.

Reshapes first dimension back to [batch_size, beam_size]. Args: tensor: Tensor to reshape of shape [batch_size*beam_size, ...] batch_size:

Generate logits for each value in the target sequence. Args: targets: target values for the output sequence. int tensor with shape

Return predicted sequence.

Initializes an instance of a LaserTagger model. Args: config: LaserTagger model configuration. num_tags: Number of differ

Computes how many times different phrases have to be added. 计算不同的短句，被添加多少次，以这些来建立优化的词汇表 Args: data_iterator: 迭代器，yield source lists

Backtracks the Longest Common Subsequence table to reconstruct the LCS. Args: table: Precomputed LCS table. source: List of source tokens.

Computes a single tag. The tag may match multiple target tokens (via tag.added_phrase) so we return the next unmatched target token.

Constructs a sparse phrase occurrence matrix. Examples are on rows and phrases on columns. Args: all_added_phrases: List of lists of a

Returns the number of examples whose added phrases are in the vocabulary. This assumes the vocabulary is created simply by selecting the `voc

Return initial state dictionary and its shape invariants. Args: initial_ids: initial ids to pass into the symbols_to_logits_fn. int

Creates a LaserTagger model.

Decodes a record to a TensorFlow example.

Lowcases the first character of the text.

Upcases the first character of the text.

Reshapes first two dimensions in to single dimension. Args: tensor: Tensor to reshape of shape [A, B, ...] Returns: Reshaped tensor of s

Returns hyperparameters for TransformerDecoder. Args: config: LaserTagger model configuration. use_tpu: Whether to train on TPUs.

Gather the top k sequences that are still alive. Args: new_seq: New sequences generated by growing the current alive sequences int3

Combine new and old finished sequences, and gather the top k sequences. Args: state: A dictionary with the current loop state. new_se

Returns the ID of the [PAD] token (or 0 if it's not in the vocab).

Returns a decoding function that calculates logits of the next tokens.

Grow alive sequences by one token, and collect top 2*beam_size sequences. 2*beam_size sequences are collected because some sequences may have rea

Returns the Longest Common Subsequence dynamic programming table.

Realizes output text corresponding to a single source text. Args: tokens: Tokens of the source text. tags: Tags indicating the edit o

Splits tokens (and the labels accordingly) to WordPieces. Args: tokens: Tokens to be split. labels: Labels (one per token) to be spli

Returns `model_fn` closure for TPUEstimator.

Apply attention mechanism to x and y. Args: x: a tensor with shape [batch_size, length_x, hidden_size] y: a tensor with shape [batch_

Combine tensor that has been split. Args: x: A tensor [batch_size, num_heads, length, hidden_size/num_heads] Returns: A tensor w

Generate continuous representation for inputs. Args: inputs: int tensor with shape [batch_size, input_length]. attention_bias: float

Creates an `input_fn` closure to be passed to TPUEstimator. 输入函数

Compute the addition score (Equation 4 in the paper).

Compute the deletion score (Equation 6 in the paper).

Compute the keep score (Equation 5 in the paper).

Return float tensor representing the padding values in x. Args: x: int tensor with any shape padding_value: int value that dtype: The d

Computes the SARI scores from the given source, prediction and targets. Args: source_ids: A 2D tf.Tensor of size (batch_size , sequence_length)

Compute the SARI score for a single prediction and one or more targets. Args: source_ids: a list / np.array of SentencePiece IDs prediction

Pad the feature vectors so that they all have max_seq_length. Args: max_seq_length: The length that features will have after padding.

Returns realized prediction for given sources.

Realize output text based on the source tokens and predicted tags. Args: tags: Predicted tags (one for each token in `self.source_tokens`).

Beam search for sequences with highest scores.

Generate logits for next potential IDs. Args: ids: Current decoded sequences. int tensor with shape [batch_size * beam_size,

Returns this object as a tf.Example.

Calculates target logits or inferred target sequences. Args: inputs: int tensor with shape [batch_size, input_length]. encoder_output

Calculate target logits or inferred target sequences. Args: inputs: int tensor with shape [batch_size, input_length]. targets: None o

Constructs a Tag object by parsing tag to tag_type and added_phrase. Args: tag: String representation for the tag which should have the fol

Initializes an instance of EditingTask. Args: sources: A list of source strings. Typically contains only one string but for sentenc

phrase_vocabulary: Iterable of phrase vocabulary items (strings). do_swap: Whether to enable the SWAP tag.

Initializes layers to build Transformer model. Args: params: hyperparameter object defining layer sizes, dropout values, etc. train:

Initializes an instance of BertExampleBuilder. Args: label_map: Mapping from tags to tag IDs. vocab_file: Path to BERT vocabulary fil

Initializes an instance of LaserTaggerPredictor. Args: tf_predictor: Loaded Tensorflow model. example_builder: BERT example builder.

Initializes an instance of LaserTagger configuration. This initializer expects both the BERT specific arguments and the Transformer d

Initialize sequence beam search. Args: symbols_to_logits_fn: A function to provide logits, which is the interface to the Transforme

Specify characteristic parameters of embedding layer. Args: vocab_size: Number of tokens in the embedding. (Typically ~32,000) hidden

Calculates the number of steps. warmup_proportion: Proportion of warmup steps. Returns: Tuple (number of steps, number of warmup steps

Return whether to continue the search loop. The loops should terminate when 1) when decode length has been reached, or 2) when the wo

Return the shape of the input tensor.

Returns a decoding function that calculates logits of the next tokens.

Beam search loop body. Grow alive sequences by a single ID. Sequences that have reached the EOS token are marked as finished. The alive and f

Returns the output of the decoder layer stacks. Args: decoder_inputs: tensor with shape [batch_size, target_length, hidden_size] enco

Return outputs of the feedforward network. Args: x: tensor with shape [batch_size, length, hidden_size] padding: (optional) If set, t

Return the output of the encoder layer stacks. Args: encoder_inputs: tensor with shape [batch_size, input_length, hidden_size] attent