Example processor that converts a dictionary of texts into tokens.

Huggingface dataset, where the dataset is loaded using the huggingface datasets.load_dataset() function.

JSON dataset, where each line of the data file contains a JSON dictionary with text fields.

Datset builder class.

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.