Generates a hash of the seed string as an unsigned int15 integer

Denormalizes an image. Args: img (torch.Tensor): Image to denormalize. mean (tuple): Mean to use for denormalization.

Convert a string separated by hyphens into a list of a given data type, replacing invalid values with None. Args: input_str

Encodes an input image tensor and quantizes the latent code. Args: x: Input image tensor of shape B C H W or B H W

Retrieves a model from a DDP wrapper, if necessary.

Retrieves a model from a DDP wrapper, if necessary.

Decodes discrete tokens back to an image. Args: tokens: Quantized indices of shape B H_Q W_Q Returns: Decode

Look up a default value belonging to a given domain and key.

Override the default train() to set the training mode to all modules except the encoder if freeze_enc is True. Args: mod

Retrieves a model from a DDP wrapper, if necessary.

Create a 1D, 2D, or 3D convolution module.

Load a model from HuggingFace hub or a given .safetensors checkpoint path.

Try to retrieve a key from the validation set config. If it does not exist, default to retrieving it from the train set config with the same

Forward computation of logits for each modality. Args: y (torch.Tensor): Decoder output. Shape (B, M, D). decoder_mod

Span masking for a sequence Args: sequence: Sequence to mask sentinel_to_id: Mapping from sentinel to id keep_prob: Proba

Render keypoints in the OpenPose format on input image. Args: img (np.array): Input image of shape (H, W, 3) with pixel values in the

Generate bbox and apply lambda correction.

Forward pass for the decoder. Args: y (torch.Tensor): Decoder input tokens. Shape (B, M, D). context (torch.Tensor):

Forward pass through embedding module, transforming sequence of ids to sequence of embeddings. Creates corresponding modality and pos

Forward pass for the encoder. Args: x (torch.Tensor): Encoder input tokens. Shape (B, N, D) where N is the number of enco

Squeeses a numpy array along a given axis if that axis is one-dimensional. Otherwise, it returns the same array. Args: array

Converts a (B C H W) tensor to numpy arrays. If B = 1, the tensor is unbatched and converted to a single image. If C = 1, the channel dim

With 50% prob, negate the value

Model can either be a single nn.Module, or a dictionary with {'model': model, 'balancer': balancer}.

Decodes a text sequence from a model dictionary. Args: mod_dict (dict): Model output dictionary. key (str): Key of the text

Modification of forward_mask_encoder adapted for generation, with support for batching

Generates a sequence of tokens from the input modalities. :param mod_dict: Dictionary of modalities. :param schedule: Schedule of mod

Crop and resize an image :param img: Image to crop and resize :param crop_coords: Coordinates of the crop (top, left, h, w) :

Initialize parts of encoder that are dependent on dimension of tokens. Should be called when setting up FourM. Args:

Visualizes a feature map using PCA. Args: features (torch.Tensor): CxHxW feature map to visualize. n_components (int): Numbe

Computes the perspective projection of a set of 3D points. Args: points (torch.Tensor): Tensor of shape (B, N, 3) containing the inpu

Plots text in a square. Args: ax (matplotlib.axes.Axes): Matplotlib axis to plot on text (str): Text to plot padding

Predict the sample at the previous timestep by reversing the SDE. Core function to propagate the diffusion process from the learned m

Decodes the vae latent representation into vae latent representaiton. Args: x: VAE latent representation clip

Zero out the parameters of a module and return it.

Autoencodes an input image tensor by encoding it, quantizing the latent code, and decoding it back to an image. Args: x:

Builds a list of chained generation schedules, where each schedule is a tuple of the form: (target_modality, schema, number of decoded tokens

Decodes the model output dictionary into a dictionary of images and text. Args: mod_dict (dict): Model output dictionary. to

Crop and resize an image :param img: Image to crop and resize :param flip: Whether to flip the image :return: Flipped image (

Initializes an empty target modality dictionary for a given domain. Used to initialize target modality dictionaries for generation.

Loads the state_dict from a checkpoint file and initializes the model with it. Renames the keys in the state_dict if necessary (e.g. when load

Aggregate a list of PIL images into a grid of images.

Optionally mask out invalid regions and concat mask to images. Useful when tokenizing simulated data that contains unlabeled regions. Args:

Make a standard normalization layer. :param channels: number of input channels. :return: an nn.Module for normalization.