Perform pooling by stride slicing the tensor along the given axis.

Perform pooling by stride slicing the tensor along the given axis.

Scikit / Keras interface to transformers' pipelines. This method will forward to __call__().

Add :obj:`arrays` to the internal storage, Will initialize the storage to the full size at the first arrays passed so that if we're b

Check if the model class is in supported by the pipeline. Args: supported_models (:obj:`List[str]` or :obj:`dict`):

Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args:

Take the output of any :obj:`ModelForQuestionAnswering` and will generate probabilities for each span to be the actual answer.

Context Manager allowing tensor allocation on the user-specified device in framework agnostic way. Returns: Context mana

r""" Instantiate an encoder and a decoder from one or two base classes of the library from pretrained model checkpoints. The

See base class.

Invert an attention mask (e.g., switches 0. and 1.). Args: encoder_attention_mask (:obj:`torch.Tensor`): An attention ma

location will always be cpu for np tensors

Tokenizes a piece of text.

Used to implement attention between consecutive chunks. Args: vectors: array of shape [batch_size, num_attention_heads,

Adds a word to the dictionary Args: word (:obj:`str`): Tthe new token/word to be added to the vocabulary. n (int

Get the index of the token in the encoded output comprising a character in the original string for a sequence of the batch.

Get the word in the original string corresponding to a character in the original string of a sequence of the batch. Can be c

Convert the inner content to tensors. Args: tensor_type (:obj:`str` or :class:`~transformers.tokenization_utils_base.Ten

Very simple data collator that simply collates batches of dict-like objects and erforms special handling for potential keys named: -

Ensure PyTorch tensors are on the specified device. Args: inputs (keyword arguments that should be :obj:`torch.Tensor`):

Returns the evaluation :class:`~torch.utils.data.DataLoader`. Subclass and override this method if you want to inject some custom be

Args: append_path: optional path to append to the tests dir path Return: The full path to the `tests` dir, so that the tests

Get token or None if not existent.

Resolve a model identifier, and a file name, to a HF-hosted url on either S3 or Cloudfront (a Content Delivery Network, or CDN). Cloudfr

Converts a list of examples into a list of features that can be directly given as input to a model. It is model-dependant and takes advantage

Get the character span corresponding to an encoded token in a sequence of the batch. Character spans are returned as a :class:`~tran

Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch. Can be called as:

Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are returned a

Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. Args: batch_index (:

Gather value of `tensors` (tensor or list/tuple of nested tensors) and convert them to numpy before concatenating them to `gathered`

splits hidden_size dim into attn_head_size and num_attn_heads

Small helper to check pseudo-equality of parsed arguments on `ArgumentParser` instances.

If tensors have different shapes, different values or a and b are not both tensors, raise a nice Assertion error.

Make a student by copying alternating layers from a teacher, save it to save_path. Args: teacher: str or PreTrainedModel if str, this will