x is a list

Yellow text

Lower text and remove punctuation, articles and extra whitespace.

Inputs: token_batch: List (length batch size) of lists of ints. batch_pad: Int. Length to pad to. If 0, pad to maximum length sequenc

Loads JSON data from filename and returns

This function returns a generator object that yields batches. The last batch in the dataset will be a partial batch. Read this to underst

Takes an URL, a filename, and the expected bytes, download the contents and returns the filename. num_bytes=None disables the file size check.

Green text

Magenta background

Reads the dataset, extracts context, question, answer, tokenizes them, and calculates answer span in terms of token indices. Note: due to toke

Red background

Red text

Turns an already-tokenized sentence string into word indices e.g. "i do n't know" -> [9, 32, 16, 96] Note any token that isn't in the word2id

Returns the total number of (context, question, answer) triples, given the data read from the SQuAD json file.

This is similar to get_batch_generator in data_batcher.py, but with some differences (see explanation in refill_batches). Inputs:

Return a mapping that maps from character locations to the corresponding token locations. If we're unable to complete the mapping e.g. becaus

Reads from original GloVe .txt file and returns embedding matrix and mappings from words to word ids. Input: glove_path: path to glove.

Read the contexts and questions from a .json file (like dev-v1.1.json) Returns: qn_uuid_data: list (length equal to dev set size) of u

Given a string e.g. '311 9 1334 635 6192 56 639', returns as a list of integers

Note: this is similar to squad_preprocess.preprocess_and_write, but: (1) We only extract the context and question information from the JSON

Pretty-print the results for one example. Inputs: word2id: dictionary mapping word (string) to word id (int) context_tokens, qn_

Adds more batches into the "batches" list. Inputs: batches: list to add batches to word2id: dictionary mapping word (string) to

This is similar to refill_batches in data_batcher.py, but: (1) instead of reading from (preprocessed) datafiles, it reads from the provided

https://github.com/tqdm/tqdm

Inputs: {context/qn}_ids: Numpy arrays. Shape (batch_size, {context_len/question_len}). Contains padding. {co

b: int, optional Number of blocks just transferred [default: 1]. bsize: int, optional Size of each block (in