Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model

input: sent_batch - list of sentences, e.g., ["abcd", "efgh"] return: know_sent_batch - list of sentences with entites embedding

Build universial encoder representations models. The combinations of different embedding, encoder, and target layers yield pretrained mo

Runs basic whitespace cleaning and splitting on a piece of text.

Mask input sequence for MLM task. args: src: a list of token ids.

Args: gpu_id: The id of GPU for single GPU mode; The id of process (and GPU) for multiprocessing distributed mode. gp

Performs invalid character removal and whitespace cleanup on text.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Adds whitespace around any CJK character.

truncate sequence pair to specific length

truncate sequence pair to specific length

Union vocab in all workers.

word_ids: batch_size, seq_length

Constructs a BertTokenizer. Args: vocab_file: Path to a one-wordpiece-per-line vocabulary file do_lower_case: Whether to l

Constructs a BasicTokenizer. Args: do_lower_case: Whether to lower case the input.