hidden_states: [seq_len, batch, hidden_size] attention_mask: [(1, 1), seq_len, seq_len]

hidden_states: [seq_len, batch, hidden_size]

hidden_states: [seq_len, batch, hidden_size] attention_mask: [(1, 1), seq_len, seq_len]

Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (

r""" Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing

r""" labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression

Forward step.

Forward step.

Forward step.

Instantiate a PreTrainedBertModel from a pre-trained model file. Download and cache the pre-trained model file if needed.

Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added token

Get the IDs for an evidence block plus the title of the corresponding document

Provide extra arguments required for merging.

Return the data currently in use.

Return world size for the data parallel group.

Get the memory buffer.

Provide extra arguments required for merging.

Provide extra arguments required for merging.

Provide extra arguments required for merging.

Get empty block and title - used in REALM pretraining

Provide extra arguments required for tasks.

Returns vocab as a dict

Initializ the memory buffer for the checkpointed activations.

Initialize model data parallel groups. Arguments: model_parallel_size: number of GPUs used to parallelize model. Let's say we h

When running with :class:`FP16_Optimizer`, ``.grad`` attributes of a model's fp16 leaves should not be regarded as truthful,

Dictionary from vocab id token to text token.

Useful for code segments that may be accessed with or without mpu initialization

Join a list of strings, handling spaces appropriately

Efficient L2 norm gradient clipping.

Customized load.

Customized load.