A sequential module that passes timestep embeddings to the children that support it as an extra input.

A residual block that can optionally change the number of channels. :param channels: the number of input channels. :param emb_channels: t

Transformer block for image-like data. First, project the input (aka embedding) and reshape to b, t, d. Then apply standard transform

An attention block that allows spatial positions to attend to each other. Originally ported from here, but adapted to the N-d case. https

A downsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determini

An upsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determinin

r""" Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs: **last_hidden_state**: ``torch.Float

A single linear layer which does a 1x1 conv

r"""A placeholder identity operator that is argument-insensitive.

Construct the embeddings from word, position and token_type embeddings.

r""" Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs: **last_hidden_state**: ``torch.Float

Transformer block for image-like data. First, project the input (aka embedding) and reshape to b, t, d. Then apply standard transform

A module which performs QKV attention and splits in a different order.

Adapted from CLIP: https://github.com/openai/CLIP/blob/main/clip/model.py

Uses a pretrained BERT tokenizer by huggingface. Vocab size: 30522 (?)

Runs basic tokenization (punctuation splitting, lower casing, etc.).

to match AttnBlock usage

to match AttnBlock usage

Defines a PatchGAN discriminator as in Pix2Pix --> see https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py

Compute SQuAD 2.0 answer class from classification and start tokens hidden states.

Compute SQuAD end_logits from sequence hidden states and start token hidden state.

Compute SQuAD start_logits from sequence hidden states.

A module which performs QKV attention. Matches legacy QKVAttention + input/ouput heads shaping

r""" Compute a single vector summary of a sequence hidden states according to various possibilities: Args of the config class: sum

Runs WordPiece tokenization.

Uses the BERT tokenizr model and add some transformer encoder layers

r""" :class:`~pytorch_transformers.BertConfig` is the configuration class to store the configuration of a `BertModel`. Argum

r""" Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs: **last_hidden_state**: ``torch.Float

r""" **masked_lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``: Labels for computing the

r""" **labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size,)``: Labels for computing the multiple choice classifica

r""" **next_sentence_label**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size,)``: Labels for computing the next sequence

r""" **masked_lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``: Labels for computing the