Tokenize data file and turn into token-ids using given vocabulary file. This function loads data line-by-line from data_path, calls the above sen

Read data from source and target files and put into buckets. Args: source_path: path to the files with token-ids for the word sequence. tar

Put attention masks on hidden using hidden_features and query.

Very basic tokenizer: split the sentence into a list of tokens.

Run a step of the intent classification model feeding the given inputs. Args: session: tensorflow session to use. encoder_inputs: lis

Create vocabulary file (if it does not exist yet) from data file. Data file is assumed to contain one sentence per line. Each sentence is tokeniz

Run a step of the joint model feeding the given inputs. Args: session: tensorflow session to use. encoder_inputs: list of numpy int v

Run a step of the tagging model feeding the given inputs. Args: session: tensorflow session to use. encoder_inputs: list of numpy int

Put attention masks on hidden using hidden_features and query.

INPUT: p :: predictions g :: groundtruth w :: corresponding words OUTPUT: filename :: name of the file where the predictions

Create model and initialize or load parameters in session.

Generate RNN state outputs with word embeddings as inputs

Get a random batch of data from the specified bucket, prepare for step. To feed data in step(..) it must be a list of batch-major vectors, while

Get a single sample data from the specified bucket, prepare for step. To feed data in step(..) it must be a list of batch-major vectors, while

run conlleval.pl perl script to obtain precision/recall and F1 score

Initialize vocabulary from file. We assume the vocabulary is stored one-item-per-line, so a file: dog cat will result in a vocabulary {"d

Convert a string to list of integers representing token-ids. For example, a sentence "I have a dog" may become tokenized into ["I", "have", "a",

Naive tokenizer: split the sentence by space into a list of tokens.