Functions — github.com/TencentARC/MindOmni

hub / github.com/TencentARC/MindOmni / functions

Functions516 in github.com/TencentARC/MindOmni

⨍ Functions 516◇ Types & classes 89 ↳ Endpoints 2

↓ 63 callersMethodto

(self, device: Union[str, torch.device])

pretrain/pipeline.py:112

↓ 48 callersMethodto

(self, device: Union[str, torch.device])

src/image_decoder/image_pipeline.py:48

↓ 46 callersMethodto

(self, device: Union[str, torch.device] = None, dtype: Union[str, torch.device] = None)

src/mindomni.py:65

↓ 42 callersMethodinfo

Print information about the annotation file. :return:

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/coco.py:78

↓ 37 callersMethodfrom_pretrained

(cls, model_path)

src/mindomni.py:52

↓ 18 callersMethodlog

(self, logs: dict[str, float], start_time: Optional[float] = None)

rl-postrain/src/open-r1-multimodal/src/open_r1/trainer/grpo_trainer.py:985

↓ 16 callersFunctionclean_text

(text, exclue_chars=['\n', '\r'])

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_jsonl.py:757

↓ 16 callersFunctionclean_text

(text, exclue_chars=['\n', '\r'])

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_ust.py:769

↓ 14 callersMethodeval

(self)

src/mindomni.py:79

↓ 12 callersMethod__init__

(self, config)

src/image_decoder/modeling_phi3.py:329

↓ 12 callersMethod__init__

(self, config)

pretrain/modeling_phi3.py:329

↓ 10 callersFunction_isArrayLike

(obj)

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/coco.py:20

↓ 8 callersMethodprocess_multi_modal_prompt

input_images: PIL.Image for vae input_llm_images: image path for vit

pretrain/processor.py:58

↓ 6 callersMethodmove_to_device

(self, data)

pretrain/pipeline.py:128

↓ 6 callersFunctionnormalize

(logit)

pretrain/train_helper/loss.py:37

↓ 6 callersMethodprocess_mllm_input

(self, mllm_inputs, target_img_size)

pretrain/processor.py:465

↓ 6 callersFunctionrepeat_kv

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

src/image_decoder/modeling_phi3.py:348

↓ 6 callersFunctionrepeat_kv

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

pretrain/modeling_phi3.py:348

↓ 6 callersFunctionrequires_grad

Set requires_grad flag for all parameters in a model.

pretrain/utils.py:34

↓ 4 callersMethod__init__

( self, transformer_config: Phi3Config, patch_size=2, in_channels=4, p

pretrain/model.py:169

↓ 4 callersMethod_get_per_token_logps

(self, model, input_ids, attention_mask, output_hidden_states=False, **custom_multimodal_inputs)

rl-postrain/src/open-r1-multimodal/src/open_r1/trainer/grpo_trainer.py:554

↓ 4 callersMethod_update_causal_mask

( self, attention_mask: torch.Tensor, input_tensor: torch.Tensor, cache_positi

pretrain/modeling_phi3.py:1097

↓ 4 callersFunctioncrop_by_max_pixels

r""" Returns the height and width of the image, downscaled to the next integer multiple of `vae_scale_factor`. Args: image (`Union[PI

pretrain/utils.py:93

↓ 4 callersFunctiondeps_list

(*pkgs)

rl-postrain/src/open-r1-multimodal/setup.py:81

↓ 4 callersFunctionextract_choice

(text)

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_jsonl.py:116

↓ 4 callersFunctionextract_choice

(text)

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_ust.py:128

↓ 4 callersMethodforward

pretrain/model.py:374

↓ 4 callersMethodfrom_pretrained

(cls, model_name)

pretrain/model.py:212

↓ 4 callersMethodgetAnnIds

Get ann ids that satisfy given filter conditions. default skips that filter :param imgIds (int array) : get anns for given imgs

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/coco.py:86

↓ 4 callersMethodget_data_info

Get data info from the dataset.

pretrain/train_helper/webdataset_.py:48

↓ 4 callersMethodget_mllm_hidden_state

(self, user_input, input_images, do_sample, temperature, max_new_tokens, only_understand=False, use_cot=False)

src/mindomni.py:86

↓ 4 callersMethodloadAnns

Load anns with the specified ids. :param ids (int array) : integer ids specifying anns :return: anns (object array) : l

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/coco.py:157

↓ 4 callersFunctionload_annotation

(data_path, is_replace_image_file=False, image_dir=None, key_list=["input_images"], debug_json_item_num=None)

pretrain/train_helper/utils.py:29

↓ 4 callersFunctionregister_lighteval_task

Registers a LightEval task configuration. - Core tasks can be added from this table: https://github.com/huggingface/lighteval/blob/main/src/light

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/evaluation.py:26

↓ 4 callersMethodupdate

Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.

src/image_decoder/scheduler.py:63

↓ 4 callersMethodupdate

Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.

pretrain/scheduler.py:82

↓ 3 callersMethod__init__

( self, transformer_config: Phi3Config, patch_size=2, in_channels=4, p

src/image_decoder/model.py:154

↓ 3 callersMethod_get_per_token_logps

( self, model, input_ids, attention_mask, pixel_values, image_

rl-postrain/src/open-r1-multimodal/src/open_r1/trainer/vllm_grpo_trainer.py:466

↓ 3 callersMethodaccumulate

Accumulate per image evaluation results and store the result in self.eval :param p: input params for evaluation :return: None

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/cocoeval.py:313

↓ 3 callersFunctionapply_rotary_pos_emb

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

src/image_decoder/modeling_phi3.py:301

↓ 3 callersFunctionapply_rotary_pos_emb

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

pretrain/modeling_phi3.py:301

↓ 3 callersFunctionget_lighteval_tasks

()

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/evaluation.py:55

↓ 3 callersMethodget_question_template

(task_type: str)

rl-postrain/src/open-r1-multimodal/src/open_r1/vlm_modules/qwen_module.py:80

↓ 3 callersFunctioniou

(box1, box2)

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_ust.py:430

↓ 3 callersFunctionis_peft_model

(model)

pretrain/train_helper/loss.py:6

↓ 3 callersFunctionmap_reward

Calculate mean average precision (mAP) reward between predicted and ground truth bounding boxes. Args: content (str): String con

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_jsonl.py:297

↓ 3 callersFunctionmap_reward

Calculate mean average precision (mAP) reward between predicted and ground truth bounding boxes. Args: content (str): String con

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_ust.py:309

↓ 3 callersFunctionmonkey_patch_qwen2_5vl_flash_attn

()

rl-postrain/src/open-r1-multimodal/src/open_r1/qwen2_5vl_monkey_patch.py:43

↓ 3 callersFunctionmonkey_patch_qwen2_5vl_forward

()

rl-postrain/src/open-r1-multimodal/src/open_r1/qwen2_5vl_monkey_patch.py:213

↓ 3 callersMethodsummarize

Compute and display summary metrics for evaluation results. Note this functin can *only* be applied on the default parameter setting

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/cocoeval.py:420

↓ 3 callersFunctionvae_encode

(vae, x, weight_dtype)

pretrain/utils.py:143

↓ 2 callersMethod_get_key_from_inputs

(self, x, key)

rl-postrain/src/open-r1-multimodal/src/open_r1/trainer/grpo_trainer.py:578

↓ 2 callersFunction_prepare_4d_causal_attention_mask_with_cache_position

Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`,

src/image_decoder/modeling_phi3.py:61

↓ 2 callersFunction_prepare_4d_causal_attention_mask_with_cache_position

Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`,

pretrain/modeling_phi3.py:61

↓ 2 callersMethod_prepare_inputs

(self, inputs)

rl-postrain/src/open-r1-multimodal/src/open_r1/trainer/grpo_trainer.py:574

↓ 2 callersMethod_update_causal_mask

( self, attention_mask: torch.Tensor, input_tensor: torch.Tensor, cache_positi

src/image_decoder/modeling_phi3.py:1097

↓ 2 callersMethodadjust_attention_for_input_images

(self, attention_mask, image_sizes)

pretrain/processor.py:421

↓ 2 callersMethodannToRLE

Convert annotation which can be polygons, uncompressed RLE to RLE. :return: binary mask (numpy 2D array)

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/coco.py:370

↓ 2 callersFunctionbuild_message

(user_input, think_content, qwen_prompt, llm_processor, model_llm, use_template, logger)

pretrain/train_helper/validate.py:11

↓ 2 callersFunctioncompute_score

(solution_str, ground_truth)

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/math.py:2

↓ 2 callersMethodcreateIndex

(self)

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/coco.py:47

↓ 2 callersMethodcreate_mask

(self, attention_mask, num_tokens_for_output_images)

pretrain/processor.py:382

↓ 2 callersMethodcreate_position

(self, attention_mask, num_tokens_for_output_images)

pretrain/processor.py:356

↓ 2 callersMethodcrop_attention_mask_for_cache

(self, attention_mask, num_tokens_for_img)

pretrain/scheduler.py:157

↓ 2 callersMethodcrop_position_ids_for_cache

(self, position_ids, num_tokens_for_img)

pretrain/scheduler.py:149

↓ 2 callersMethodcropped_pos_embed

Crops positional embeddings for SD3 compatibility.

src/image_decoder/model.py:261

↓ 2 callersMethodcropped_pos_embed

Crops positional embeddings for SD3 compatibility.

pretrain/model.py:287

↓ 2 callersFunctiondetection_score

(content, sol, iou_threshold=0.5, alpha=0.7, beta=0.0, gamma=0.3)

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_jsonl.py:431

↓ 2 callersFunctiondetection_score

(content, sol, iou_threshold=0.5, alpha=0.7, beta=0.0, gamma=0.3)

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_ust.py:443

↓ 2 callersMethodevaluate

Run per image evaluation on given images and store results (a list of dict) in self.evalImgs :return: None

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/cocoeval.py:119

↓ 2 callersMethodevict_previous_layer

Moves the previous layer cache to the CPU

src/image_decoder/scheduler.py:28

↓ 2 callersMethodevict_previous_layer

Moves the previous layer cache to the CPU

pretrain/scheduler.py:40

↓ 2 callersMethodforward

(self, x, timestep, padding_latent=None, past_key_values=None, return_past_key_values=True, offload_model: boo

src/image_decoder/model.py:321

↓ 2 callersFunctionforward_func

(output_hidden_states)

rl-postrain/src/open-r1-multimodal/src/open_r1/vlm_modules/gen_kl.py:116

↓ 2 callersMethodgetImgIds

Get img ids that satisfy given filter conditions. :param imgIds (int array) : get imgs for given ids :param catIds (int array

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/coco.py:136

↓ 2 callersFunctionget_1d_sincos_pos_embed_from_grid

embed_dim: output dimension for each position pos: a list of positions to be encoded: size (M,) out: (M, D)

src/image_decoder/model.py:110

↓ 2 callersFunctionget_1d_sincos_pos_embed_from_grid

embed_dim: output dimension for each position pos: a list of positions to be encoded: size (M,) out: (M, D)

pretrain/model.py:115

↓ 2 callersMethodget_example

(self, index)

pretrain/train_helper/concatedataset.py:37

↓ 2 callersMethodget_example

(self, index)

pretrain/train_helper/subdata.py:60

↓ 2 callersMethodget_example

(self, index)

pretrain/train_helper/data.py:116

↓ 2 callersMethodloadRes

Load result file and return a result api object. :param resFile (str) : file name of result file :return: res (obj)

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/pycocotools/coco.py:262

↓ 2 callersFunctionmake_conversation_image

(example)

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo.py:160

↓ 2 callersFunctionmean_flat

Take torche mean over all non-batch dimensions.

pretrain/train_helper/loss.py:170

↓ 2 callersFunctionmonkey_patch_torch_load

()

rl-postrain/src/open-r1-multimodal/src/open_r1/qwen2_5vl_monkey_patch.py:225

↓ 2 callersMethodmove_to_device

(self, data)

src/image_decoder/image_pipeline.py:64

↓ 2 callersFunctionnormalize

(logit)

rl-postrain/src/open-r1-multimodal/src/open_r1/vlm_modules/gen_kl.py:16

↓ 2 callersFunctionod_reward

Calculate reward for object detection task by comparing predicted and ground truth answers. Args: content (str): Model's predict

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_jsonl.py:351

↓ 2 callersFunctionod_reward

Calculate reward for object detection task by comparing predicted and ground truth answers. Args: content (str): Model's predict

rl-postrain/src/open-r1-multimodal/src/open_r1/grpo_ust.py:363

↓ 2 callersMethodpad_input_ids

(self, input_ids, image_sizes, llm_mode=False)

pretrain/processor.py:428

↓ 2 callersMethodpatch_multiple_resolutions

(self, latents, padding_latent=None, is_input_images:bool=False)

pretrain/model.py:311

↓ 2 callersMethodpost_model_init

(self, model, processing_class)

rl-postrain/src/open-r1-multimodal/src/open_r1/vlm_modules/vlm_module.py:18

↓ 2 callersMethodprocess_image

(self, image_file)

pretrain/train_helper/concatedataset.py:31

↓ 2 callersMethodprocess_image

(self, image_file)

pretrain/train_helper/subdata.py:54

↓ 2 callersMethodprocess_image

(self, image_file)

pretrain/train_helper/data.py:110

↓ 2 callersMethodprocess_mllm_input

(self, context_hidden_state, target_img_size)

src/image_decoder/processor.py:178

↓ 2 callersFunctionrotate_half

Rotates half the hidden dims of the input.

src/image_decoder/modeling_phi3.py:293

↓ 2 callersFunctionrotate_half

Rotates half the hidden dims of the input.

pretrain/modeling_phi3.py:293

↓ 2 callersMethodselect_reward_func

(func: str, task_type: str)

rl-postrain/src/open-r1-multimodal/src/open_r1/vlm_modules/qwen_module.py:249

↓ 2 callersFunctionstrip_string

(string)

rl-postrain/src/open-r1-multimodal/src/open_r1/utils/math.py:158

↓ 2 callersMethodunpatchify

x: (N, T, patch_size**2 * C) imgs: (N, H, W, C)

src/image_decoder/model.py:249

next →1–100 of 516, ranked by callers