A map of modules from TF to PyTorch. This time I use a map to keep the PyTorch model as identical to the original PyTorch model as possible.

A map of modules from TF to PyTorch. I use a map to keep the PyTorch model as identical to the original PyTorch model as possible.

Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code bar

cache hidden states into memory.

Clean current folder remove saved model and training log

Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms.

Loads a data file into a list of `InputBatch`s.

Loads a data file into a list of `InputBatch`s `cls_token_at_end` define the location of the CLS token: - False (Default, BERT/

计算每种标签预测正确的个数(对应精确率、召回率计算公式上的tp)，用于后面精确率以及召回率的计算

Create 3D attention mask from a 2D tensor mask. Args: from_tensor: 2D or 3D Tensor of shape [batch_size, from_seq_length, ...]. to_mask: in

:param is_train: Flag, True for train batch :param batch: list train/evaluate data :return: structured data to feed

Creates `TrainingInstance`s for a single document.

Creates causal attention mask. Float mask where 1.0 indicates masked, 0.0 indicates not-masked. Args: qlen: TODO Lysandr

Creates the predictions for the masked LM objective.

Creates a classification model.

Creates a classification model.

Creates a classification model.

Get the vocab file and casing info from the Hub module.

Create `TrainingInstance`s from raw text.

使用维特比算法对给定观测序列求状态序列， 这里就是对字组成的序列,求其对应的标注。 维特比算法实际是用动态规划解隐马尔可夫模型预测问题，即用动态规划求概率最大路径（最优路径） 这时一条路径对应着一个状态序列

Looks up words embeddings for id tensor. Args: input_ids: int32 Tensor of shape [batch_size, seq_length] containing word ids. vocab_s

TPU and GPU embedding_lookup function.

Performs various post-processing on a word embedding tensor. Args: input_tensor: float Tensor of shape [batch_size, seq_length, embedding

Returns a dictionary containing the encoded sequence or sequence pair and additional informations: the mask for sequence classificati

Flattens (potentially nested) a tuple/dictionary/list to a list.

Constructs a `BertConfig` from a Python dictionary of parameters.

Constructs a `Config` from a Python dictionary of parameters.

Constructs a `BertConfig` from a json file of parameters.

Gathers the vectors at the specific positions over a minibatch.

GELU activation https://arxiv.org/abs/1606.08415 https://github.com/huggingface/pytorch-openai-transformer-lm/blob/master/model_pytorch.p

Maps a string to a Python function, e.g., "relu" => `tf.nn.relu`. Args: activation_string: String name of the activation function. Returns:

Compute the union of the current variables and checkpoint variables.

Gets output of the embedding lookup (i.e., input to the transformer). Returns: float Tensor of shape [batch_size, seq_length, hidden_size]

Gets entities from sequence. note: BIO Args: seq (list): sequence of labels. Returns: list: list of (chunk_type, chun

Gets entities from sequence. note: BIOS Args: seq (list): sequence of labels. Returns: list: list of (chunk_type, chu

Project the tokenized prediction back to the original text.

Given a URL, look for the corresponding dataset in the local cache. If it's not there, download it. Then return the path to the cached file.

Get loss and log probs for the masked LM.

Generate hidden states mask, and optionally an attention mask.

Get loss and log probs for the next sentence prediction.

Returns: float32 Tensor in shape [len, bsz, d_model]. The last layer hidden representation of XLNet.

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the

Finds all of the tensors in graph that are unreachable from outputs.

训练并评估hmm模型

将targets中的数转化为在[T*T]大小序列中的索引,T是标注的种类

Creates an `input_fn` closure to be passed to TPUEstimator.

Check that tags have a valid IOB format. Tags in IOB1 format are converted to IOB2.

IOB -> IOBES

Runs layer normalization followed by dropout.

Load configuration of the model parameters are stored in json format

加载json文件 :param json_path: :param file_name: :return:

Load tf checkpoints in a pytorch model.

Loads a vocabulary file into a dictionary.

Loads a vocabulary file into a dictionary.

Loads a vocabulary file into a dictionary.

r""" Computes log probabilities for all :math:`n\_classes` From: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/adaptive.py

calculate crf loss :param project_logits: [1, num_steps, num_tags] :return: scalar loss