ID is used to identify vocab items

ValueProvider is a simple interface for tensors responses without the baggage

Basic is a BasicTokenizer that runs Runs basic tokenize (punctuation splitting, lower casing, etc.).

Bert is a model that translates features to values from an exported model. It processes as follows: Pipeline: text -> FeatureFactory -> TensorFunc ->

BertOption configures a BERT model

Dict is a container for tokens NOTE: python uses an OrderedDict, unsure of implications

Estimator matches the tf, p

Feature is an input feature for a BERT model. Maps to extract_features.InputFeature in ref-impl

FeatureFactory will create features with the supplied tokenizer and sequence length

FeatureTensorFunc translates features to tensors

Full is a FullTokenizer which comprises of a Basic & Wordpiece tokenizer

InputFunc matches feeds in sessions

ModelFunc the returned params match fetches & targets from the API

Option alter the behavior of the tokenizer TODO add tests for these behavior changes

Predictor creates tensors for prediction

Provider is an interface for exposing a vocab

TensorInputFunc maps tensors to an estimator.InputFunc in the Predict pipeline

Tokenizer is an interface for chunking a string into it's tokens as per the BERT implematation

VocabTokenizer comprises of a Tokenizer and VocabProvider

Wordpiece is a tokenizer that breaks tokens into subword units based on a supplied vocabulary https://arxiv.org/pdf/1609.08144.pdf Section 4.1 for det

engine is a simple semantic search engine for demonstrating using a BERT model It is not exported because it is not meant to be used outside of demons

TODO batch from flag