Applies radial or OpenCV distortion to the given 2D points. Args: extra_params (torch.Tensor or numpy.ndarray): Distortion parameter

Generate raw model output for images and instructions Args: images (PIL.Image or List[PIL.Image]): Input image(s

Generate raw model output for images and instructions Args: images (PIL.Image or List[PIL.Image]): Input image(s

Extract bounding boxes, points, and thinking process from model output Args: output_text (str): Raw output text

Extract bounding boxes, points, and thinking process from model output Args: output_text (str): Raw output text

Convert input to torch tensor if it's not already one.

Detect objects in an image based on a query using YOLO model Args: image: Input image query: Text qu

Detect objects in an image based on a query using YOLO model Args: image: Input image query: Text qu

Check if YOLO should be used for the given query Args: query (str): Text query describing what to detect

Check if YOLO should be used for the given query Args: query (str): Text query describing what to detect

Generate response from the model Args: image (PIL.Image): Input image prompt (str): Text prompt

Generate response from the model Args: image (PIL.Image): Input image prompt (str): Text prompt

Implementation of the forward pass through the DPT head. This method processes a specific chunk of frames from the sequence.

Apply basic activation function to pose parameters. Args: pose_enc: Tensor containing encoded pose parameters act_type: Acti

Extract JSON content from model response using triple quotes Args: response_text (str): Model response text

Extract JSON content from model response using triple quotes Args: response_text (str): Model response text

Apply inverse log transform: sign(y) * (exp(|y|) - 1) Args: y: Input tensor Returns: Transformed tensor

Applies 1D rotary position embeddings along one dimension. Args: tokens: Input token features. positions: Position in

Apply positional embedding to tensor x.

Helper function to get camera parameters based on camera type. Args: fidx: Frame index intrinsics: Camera intrinsic paramete

Extract predicted category from model response

Process a single query frame for track prediction. Args: query_index: Index of the query frame images: Tensor of shape [S, 3

Get batch model responses Args: instructions: List of instructions Returns: list: List o

Answer a question about an image Args: image: Input image question: Text question

Answer a question about an image Args: image: Input image question: Text question

Answer a question about an image Args: image: Input image question: Text question

r"""Sample a tensor using bilinear interpolation `bilinear_sampler(input, coords)` samples a tensor :attr:`input` at coordinates :attr:`coord

r"""Sample a tensor using bilinear interpolation `bilinear_sampler(input, coords)` samples a tensor :attr:`input` at coordinates :attr:`coord

Compute the scores, i.e., the standard deviation of the 2D similarity heatmaps, given the query point features and reference frame feature ma

Count objects in an image based on a query Args: image: Input image query: Text query describing wha

Count objects in an image based on a query Args: image: Input image query: Text query describing wha

Count objects in an image based on a query Args: image: Input image query: Text query describing wha

Custom interpolate to avoid INT_MAX issues in nn.functional.interpolate.

Detect objects in an image based on a query Args: image: Input image query: Text query describing wh

Detect objects in an image based on a query Args: image: Input image query: Text query describing wh

Detect objects in an image based on a query Args: image: Input image query: Text query describing wh

Detect objects in a batch of images Args: images: List of input images queries: List of text queries

Extract answer for math tasks

Extract answer for QA tasks Args: output_text (str): Raw output text from the model Returns

Extract answer for QA tasks Args: output_text (str): Raw output text from the model Returns

Generate an image based on a query Args: refer_image_path: Path to the reference image image_prompt:

Generate segmentation masks for given image, bounding boxes and points Args: image (PIL.Image): Input image

Generate segmentation masks for given image, bounding boxes and points Args: image (PIL.Image): Input image

This function generates a 1D positional embedding from a given grid using sine and cosine functions. Args: - embed_dim: The embedding di

This function generates a 1D positional embedding from a given grid using sine and cosine functions. Args: - embed_dim: The embedding di

this will perform the index select, cat the tensors, and provide the attn_bias from cache

Initialize feature extractors that can be reused based on a method string. Args: max_query_num: Maximum number of keypoints to extra

Iteratively undistort the normalized tracks using the given distortion parameters. Args: params (torch.Tensor or numpy.ndarray): Dis

A quick start function to load and preprocess images for model input. This assumes the images should have the same shape for easier batching,

This function generates a 1D positional embedding from a given grid using sine and cosine functions. Args: - embed_dim: The embedding di

This function processes images for inference. Args: images (torch.Tensor): The images to be processed with shape S x 3 x

Process a single image with given instruction Args: image (PIL.Image): Input image instruction (str)

NumPy clone of ``project_3D_points``. Parameters ---------- points3D : (N,3) world-space points. extrinsics : (B

r"""Sample spatial features `sample_features4d(input, coords)` samples the spatial features :attr:`input` represented by a 4D tensor :math:`(

r"""Sample spatial features `sample_features4d(input, coords)` samples the spatial features :attr:`input` represented by a 4D tensor :math:`(

Segment objects in an image based on a query Args: image: Input image query: Text query describing w

Segment objects in an image based on a query Args: image: Input image query: Text query describing w

Segment objects in an image based on a query Args: image: Input image query: Text query describing w

Segment objects in a batch of images Args: images: List of input images queries: List of text querie

Processes specialized tokens with shape (1, 2, X, C) for multi-frame processing: 1) Uses the first position (index=0) for the first frame onl

Reorder tensors along a specific dimension according to the given order. Args: tensors: List of tensors to reorder order: Te

Build the patch embed layer. If 'conv', we use a simple PatchEmbed conv layer. Otherwise, we use a vision transformer.

Initializes the position generator with an empty cache.

Build a pyramid of feature maps from the input. fmaps: Tensor (B, S, C, H, W) num_levels: number of pyramid levels (each dow

Augment tracking for frames with insufficient visibility. Args: pred_tracks: List of numpy arrays containing predicted tracks.

Computes frequency components for rotary embeddings. Args: dim: Feature dimension (must be even). seq_len: Maximum se

Determine the task type based on the instruction Args: instruction (str): Text instruction or query

Determine the task type based on the instruction Args: instruction (str): Text instruction or query

Get model response for a single instruction

Process frame attention blocks. We keep tokens in shape (B*S, P, C).

Process global attention blocks. We keep tokens in shape (B, S*P, C).

Performs feature rotation by splitting and recombining feature dimensions. Args: x: Input tensor to rotate. Returns:

Returns torch.sqrt(torch.max(0, x)) but with a zero subgradient where x is 0.

Process network output to extract 3D points and confidence values. Args: out: Network output tensor (B, C, H, W) activation:

Activate pose parameters with specified activation functions. Args: pred_pose_enc: Tensor containing encoded pose parameters [transl