`bool`: Whether or not the slow tokenizer can be saved. Usually for sentencepiece based slow tokenizer, this can only be `True` if th

Checkpoint a model or part of the model. This has been directly copied from torch.utils.checkpoint.

Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms. Args: ou

Clips gradient norm of an iterable of parameters. This is adapted from torch.nn.utils.clip_grad.clip_grad_norm_ and added functionality to ha

dictionary mapping command tokens to ids for tokenizer

list (or iterable) of all command tokens for tokenizer

add cmdline flags for configuring datasets

Converts a sequence of ids in BPE tokens using the vocab.

`tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer.

Set the groups to none.

@param text: Text to encode. @param linebreak: Whether to encode newline (\n) in text. @param whitespaces: Whether to encode

Build filename's path if it does not already exists.

`Optional[int]`: Id of the end of sentence token in the vocabulary. Returns `None` if the token has not been set.

Return the url and etag (which may be ``None``) stored for `filename`. Raise ``EnvironmentError`` if `filename` or its stored metadata do not