Args: num_patches: Number of patches in the image, None for 1D inputs. max_temporal_length: Maximum length of the tem

Augment the dataset (corresponds to metadata) with the given augment function Args: metadata: list of video metadata - s

Build a data augmentation (composed of augment_fn and augment_msg) that will be applied to the dataset if None, no data augmentation

Builds a download plan for the specified split. Returns a dictionary with the following structure: { yout

Create all pytorch nn.Modules here.

Check if the attributes are valid.

Check if the config is valid

Converts the state_dict to the format expected by the compiled model.

Compute the average depth within the masked region.

Create binary mask for center crop of the full frame. Args: H: Frame height W: Frame width Returns:

Convert a dictionary to a string without quotes.

NOTE: The RealEstate10k dataset is a collection of YouTube videos, and downloading them should be done with caution to ensure that the datase

Given a video metadata for a RealEstate10K scene, compute the maximum angle (degrees) of rotation of the camera within the video.

Computes the optical flow between the corresponding frames of two video batches.

Computes a static score given the optical flow. Args: flow: Optical flow of shape (B, 2, H, W). Returns:

Generate random noise levels for training.

Initialize Video Depth Anything model.

Interpolate the corresponding frames of videos1 and videos2.

A general method for frame interpolation. Given a video of any length > 2, when the left and right key frames are known, it (iteratively, if

Load EMA weights to state dict.

Log miscallany during validation/test step.

Log videos during validation/test step.

Converts the number of tokens to the number of frames.

Apply the RMSNorm normalization to the input tensor. Args: x (torch.Tensor): The input tensor. Returns:

Predict a sequence given context tokens at the beginning, using sliding window if necessary. Args ---- context: torch

Preprocesses videos to {self.resolution}x{self.resolution} resolution

Converts the raw camera poses to concat-flattened intrinsics and extrinsics. Args: external_cond (torch.Tensor): Raw came

Randomly crop the video to a random temporal length, if not provided. Same length across all GPUs.

Reads a txt file containing a video path, a list of timestamps, and a tensor of camera poses.

Convert video tensor to list of numpy frames. Args: video_tensor: Tensor of shape (T, C, H, W) or (B, T, C, H, W), range

Rearrange input tensor to be compatible with transformer blocks, if necessary. Args: x: Input tensor of shape (B * T, C,

Rescale and crop the images to the specified shape. Args: images (torch.Tensor): images tensor of shape (*batch, c, h, w)

Run the blocks (except up/downsampling blocks) for a given level. Gradient checkpointing is used optionally, with self.checkpoints[i_

The unified sampling method, with length up to maximum token size. context of length can be provided along with a mask to achieve con

Check if the dataset should be downloaded

Converts the state_dict if self.diffusion_model is compiled, to uncompiled.

Rearrange input tensor back to its original shape, if necessary. Args: x: Input tensor of shape (B, T * H * W, C).

Note: FVD, IS, REAL_IS: (B, C) (I3D features) FVMD: (B, T, C, H, W), [0, 1] FID, LPIPS, MSE, SSIM, PSNR:

Update metric states with new predictions and conditions. Args: preds: Predicted frames of shape (B, T, C, H, W)

Update metric states with new predictions and conditions. Args: preds: Predicted frames of shape (B, T, C, H, W)

Update backbone config with camera pose conditioning

Update all metrics during validation/test step.

Augment the dataset

Wrapper for grid_sample, uses pixel coordinates

Wrapper for grid_sample, uses pixel coordinates

Broadcasts the result of func() from rank 0 to all other processes. If distributed is not initialized, returns the result of func() directly.

Build an experiment instance based on registry :param cfg: configuration file :param logger: optional logger for the experiment :para

Build metadata for the dataset and save it in metadata_dir This may vary depending on the dataset. Default: ```

Build a transform that will be applied to each video frame. (e.g. resize, center crop)