:param sess: session to run the batch :param is_train: a flag indicate if it is a train batch :param batch: a dict containing

Create a mapping (item to ID / ID to item) from a dictionary. Items are ordered by decreasing frequency.

Load sentences. A line must contain at least a word and its tag. Sentences are separated by empty lines.

Prepare the dataset. Return a list of lists of dictionaries containing: - word indexes - word char indexes

Check and update sentences tagging scheme to IOB2. Only IOB1 and IOB2 schemes are accepted.

Create a dictionary and a mapping of words, sorted by frequency.

Create a dictionary of items from a list of list of items.

:param logits: [batch_size, num_steps, num_tags]float32, logits :param lengths: [batch_size]int32, real length of each sequence

Segment text with jieba features are represented in bies format s donates single word

IOBES -> IOB

Load configuration of the model parameters are stored in json format

:param idcnn_inputs: [batch_size, num_steps, emb_size] :return: [batch_size, num_steps, cnn_output_width]

Augment the dictionary with words that have a pretrained embedding. If `words` is None, we add every word that has a pretrained embed

:param lstm_inputs: [batch_size, num_steps, emb_size] :return: [batch_size, num_steps, 2*lstm_dim]

Clean current folder remove saved model and training log

:param is_train: Flag, True for train batch :param batch: list train/evaluate data :return: structured data to feed

:param char_inputs: one-hot encoding of sentence :param seg_inputs: segmentation feature :param config: wither use segmentati

Convert full-width character to half-width one

Take sentence data and return an input for the training or the evaluation function.

Check that tags have a valid IOB format. Tags in IOB1 format are converted to IOB2.

IOB -> IOBES

Load word embedding from pre-trained file embedding size must match

calculate crf loss :param project_logits: [1, num_steps, num_tags] :return: scalar loss

calculate lstm decode loss :param project_logits: :param lengths: :param name: :return: loss

Make folders for training and evaluation

Print configuration of the model

hidden layer between lstm layer and logits :param lstm_outputs: [batch_size, num_steps, emb_size] :return: [batch_size, num_

:param lstm_outputs: [batch_size, num_steps, emb_size] :return: [batch_size, num_steps, num_tags]

Save configuration of the model parameters are stored in json format

Create a dictionary and a mapping of tags, sorted by frequency.

Run perl script to evaluate model

Replace every digit in a string by a zero.

Convert conll data to text

Take sentence data and return an input for the training or the evaluation function.

Cut text to sentences

:param sess: session to run the model :param data: list of data :param id_to_tag: index to tag name :return: evalua

Replace singletons by the unknown word with a probability p.

Load mappings from the file

Save mappings and invert mappings

tags classes can switch (IOB / IOBES) :param tags: :return: