PredictValues will run the BERT model on the provided texts. The returned values are in the same order as the provided texts.

GetID will return the ID of the token in the vocab. Will be negative if it doesn't exists

tokenizeWhitespace splits text into tokeens by whitespace, per python semantics empty strings are not included

Int32 int32 representation of an ID

Count will return the number of tokens in the feature by counting the mask bits

Feature will create a single feature from the factory ID creation is thread safe and incremental

Features will tokenize a text

LongestSubstring returns the longest token that is a substring of the token

NewBert will create a new default BERT model from the exported model and vocab. Generally used for producing embeddings

NewTokenizer returns a new FullTokenizer Use Option array to modify default behavior

IsChinese validates that rune c is in the CJK range according to BERT spec

IsControl checks wher rune c is a BERT control character

IsPunctuation checks wher rune c is a BERT punctuation character

IsWhitespace checks whether rune c is a BERT whitespace character

TODO extract this into a reusable package

padChinese will add space padding around all CJK chars This implementation matches BasicTokenizer._tokenize_chinese_chars

SequenceFeature will take a sequence string and build features for the model from it

truncate uses heuristic of trimming seq with longest len until seqlen satisfied

Add will add an item to the vocabulary, is not thread-safe

Features will create multiple features with incremental IDs

NewBasic returns a basic tokenizer. Method is supplied to match constructor of other tokenizers

NewWordpiece returns a WordpieceTokenizer with the default settings. Generally should be used in a FullTokenizer

* // GetToken will get a token by the ID, returns the mepty string if ID does not exist func (v Dict) GetToken(id ID) token { for k, v := range v.tok

Tokenize will segment a texxt into individual tokens. Follows algorithm from ref-imp Clean, PadChinese, Whitespace Split, Lower?, SplitPunc, Whitespac

Tokenize will segment the text into subword tokens from the supplied vocabulary NOTE: This implementation does not EXACTLY match the ref-impl and beha

WithModelFunc applies the given model func, used when outputs do not match the default

WithSeqLen applies the seqlen, should match max_seq_len from trained model

unzips archive and flattens directory structure

readCSV will read a CSV file and parse the records into maps keyed by column headers, requires a column with the header "text"

FromFile will read a newline delimited file into a Dict

New iwll return a a covab dict from the given tokens, IDs will match index

NewBertClassifier returns a model configured for classification after being fine-tuned with run_classification.py

NewEmbeddings returns a pre-trained model for text embeddings

NewPredictor creates a new Predictor in lieu of a full estimator

Predictor will apply fn to the estimator model

Print is a utility for printing the operations in a saved model

SetMaxWordChars will set the max chars for a word to be tokenized, generally this should be congfigured through the FullTokenizer

SetUnknownToken will set the , generally this should be congfigured through the FullTokenizer

***** New Tests *****/

***** Ported Tests *****/

Tokenize will tokenize the input text First basic is applited, then wordpiece on the tokens froms basic

Vocab returns the vocub used for this tokenizer

WithFeatureFactory replaces the default feature factory

WithInputFunc updates the input func, used if input tensors vary from defaults

WithLower will lowercase all input if set to true, or skip lowering if false NOTE: kink from reference implementation is that lowering also strips acc

WithMaxChars sets the maximum len of a token to be tokenized, if longer will be labeled as unknown

WithTokenizer applies the given tokenizer to the model

WithUnknownToken will alter the unkown token from default [UNK]

* 1. Download base model 2. Fine tune w/ run_classifier 3. export_classifier $MODEL_DIR $EXPORT_DIR 2 4. MODEL_PATH=$EXPORT_DIR go run main.go */