Functions — github.com/OpenSparseLLMs/Linear-MoE

hub / github.com/OpenSparseLLMs/Linear-MoE / functions

Functions1,498 in github.com/OpenSparseLLMs/Linear-MoE

⨍ Functions 1,498◇ Types & classes 227

↓ 1 callersMethodapply_input_jitter

Add noise to the input tensor. Refer to https://arxiv.org/abs/2101.03961. Args: input (Tensor): Input tensor. Re

linear_moe/model/mixtral/moe/router.py:204

↓ 1 callersMethodapply_input_jitter

Add noise to the input tensor. Refer to https://arxiv.org/abs/2101.03961. Args: input (Tensor): Input tensor. Re

linear_moe/model/qwen2/moe/router.py:303

↓ 1 callersMethodapply_input_jitter

Add noise to the input tensor. Refer to https://arxiv.org/abs/2101.03961. Args: input (Tensor): Input tensor. Re

linear_moe/model/deepseek_v2/moe/router_old.py:217

↓ 1 callersMethodapply_input_jitter

Add noise to the input tensor. Refer to https://arxiv.org/abs/2101.03961. Args: input (Tensor): Input tensor. Re

linear_moe/model/deepseek_v2/moe/router.py:303

↓ 1 callersMethodapply_load_balancing_loss

Applies auxiliary loss to the MoE layer. Args: probs (torch.Tensor): The probs output by the router for each token. [num_tokens,

linear_moe/model/qwen2/moe/router.py:248

↓ 1 callersMethodapply_load_balancing_loss

Applies auxiliary loss to the MoE layer. Args: loss_func (callable): The loss function to be used. probs (torch.Tenso

linear_moe/model/deepseek_v2/moe/router_old.py:171

↓ 1 callersMethodapply_load_balancing_loss

Applies auxiliary loss to the MoE layer. Args: probs (torch.Tensor): The probs output by the router for each token. [num_tokens,

linear_moe/model/deepseek_v2/moe/router.py:248

↓ 1 callersMethodapply_z_loss

Encourages the router's logits to remain small to enhance stability. Please refer to the ST-MoE paper (https://arxiv.org/pdf/2202.08906.pdf) f

linear_moe/model/mixtral/moe/router.py:189

↓ 1 callersMethodapply_z_loss

Encourages the router's logits to remain small to enhance stability. Please refer to the ST-MoE paper (https://arxiv.org/pdf/2202.08906.pdf) f

linear_moe/model/qwen2/moe/router.py:279

↓ 1 callersMethodapply_z_loss

Encourages the router's logits to remain small to enhance stability. Please refer to the ST-MoE paper (https://arxiv.org/pdf/2202.08906.pdf) f

linear_moe/model/deepseek_v2/moe/router_old.py:196

↓ 1 callersMethodapply_z_loss

Encourages the router's logits to remain small to enhance stability. Please refer to the ST-MoE paper (https://arxiv.org/pdf/2202.08906.pdf) f

linear_moe/model/deepseek_v2/moe/router.py:279

↓ 1 callersMethodaux_loss_load_balancing

Apply loss-based load balancing to the logits tensor. Args: logits (torch.Tensor): The logits tensor. Returns:

linear_moe/model/mixtral/moe/router.py:150

↓ 1 callersMethodaux_loss_load_balancing

Apply loss-based load balancing to the logits tensor. Args: logits (torch.Tensor): the logits tensor after gating, shape:

linear_moe/model/qwen2/moe/router.py:225

↓ 1 callersMethodaux_loss_load_balancing

Apply loss-based load balancing to the logits tensor. Args: logits (torch.Tensor): The logits tensor. Returns:

linear_moe/model/deepseek_v2/moe/router_old.py:151

↓ 1 callersMethodaux_loss_load_balancing

Apply loss-based load balancing to the logits tensor. Args: logits (torch.Tensor): the logits tensor after gating, shape:

linear_moe/model/deepseek_v2/moe/router.py:225

↓ 1 callersFunctionbeam_search

Perform beam search to generate sequences. Args: model (torch.nn.Module): The model used for beam search. prompts (List[List

linear_moe/generation/api.py:264

↓ 1 callersFunctionbeam_search_and_return_on_first_stage

(model, tokens, lengths,

linear_moe/generation/generation.py:354

↓ 1 callersFunctionbias_gelu

(y, bias)

linear_moe/model/common_modules/activations.py:195

↓ 1 callersFunctionbias_gelu_bwd

Assume that y has shape (B, D) and bias has shape (D)

linear_moe/model/common_modules/activations.py:204

↓ 1 callersFunctionbuild_dataset

( data_path: str, tokenizer: transformers.PreTrainedTokenizer, sequence_length: int,

toolkits/pretrain_data_preprocessing/qwen_hf_preprocess_datasets.py:8

↓ 1 callersMethodbuild_inputs_with_special_tokens

(self, token_ids_0, token_ids_1=None)

linear_moe/tokenizer/tokenization_yi.py:171

↓ 1 callersMethodcenter_crop

Center crop an image. If the image is too small to be cropped to the size given, it will be padded (so the returned result will alway

linear_moe/data/llava/cvcuda_image_processing_clip.py:162

↓ 1 callersFunctioncheck_hf_mg_forward

(hfmodel, mgmodel, mgargs)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py:765

↓ 1 callersFunctioncheck_mg_eg_forward

(mgmodel, hgmodel, mgargs)

toolkits/model_checkpoints_convertor/deepseek/hf2mcore_deepseek_v2_moe.py:425

↓ 1 callersFunctioncheck_mg_eg_forward

(mgmodel, hgmodel, mgargs)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_moe.py:405

↓ 1 callersFunctionclean_text

(raw)

toolkits/pretrain_data_preprocessing/preprocess_wudao2.py:11

↓ 1 callersFunctionclean_text

(raw)

toolkits/pretrain_data_preprocessing/clean_raw_text.py:12

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/falcon40b/checkpoint_reshaping_and_interoperability.py:602

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

(mgmodel, hgmodel, args)

toolkits/model_checkpoints_convertor/deepseek/hf2mcore_deepseek_v2_moe.py:158

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/llama/hf2mcore_70b.py:551

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/llama/hf2megatron.py:748

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

(mgmodel, hgmodel, args)

toolkits/model_checkpoints_convertor/llama/hf2mcore.py:330

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/mistral/hf2megatron.py:698

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

(mgmodel, hgmodel, args)

toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:288

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/mistral/hf2mcore_mixtral.py:660

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/baichuan/checkpoint_reshaping_and_interoperability.py:656

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/falcon/checkpoint_reshaping_and_interoperability.py:599

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/starcoder/checkpoint_reshaping_and_interoperability.py:324

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.0.py:679

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

(mgmodel, hgmodel, args)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_moe.py:158

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

(mgmodel, hfmodel, args)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py:307

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

(mgmodel, hgmodel, args)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_mha.py:120

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_gqa.py:548

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

(mgmodel, hgmodel, args)

toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:322

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.5.py:763

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/bloom/checkpoint_reshaping_and_interoperability.py:302

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/bloom/reward_model_to_megatron.py:302

↓ 1 callersFunctionconvert_checkpoint_from_megatron_to_transformers

Convert NVIDIA Megatron-LM checkpoint to HuggingFace Transformers checkpoint. This handles Megatron checkpoints with different tensor paralle

toolkits/model_checkpoints_convertor/baichuan2/checkpoint_reshaping_and_interoperability.py:628

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/falcon40b/checkpoint_reshaping_and_interoperability.py:306

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(hgmodel, mgmodel, args)

toolkits/model_checkpoints_convertor/deepseek/hf2mcore_deepseek_v2_moe.py:219

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(args)

toolkits/model_checkpoints_convertor/llama/hf2mcore_70b.py:271

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/llama/hf2megatron.py:362

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(mgmodel, hgmodel, args, hf_config)

toolkits/model_checkpoints_convertor/llama/hf2mcore.py:377

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/mistral/hf2megatron.py:352

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(mgmodel, hgmodel, args, hf_config)

toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:326

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(args)

toolkits/model_checkpoints_convertor/mistral/hf2mcore_mixtral.py:353

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/baichuan/checkpoint_reshaping_and_interoperability.py:346

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/glm/checkpoint_reshaping_and_interoperability.py:271

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/falcon/checkpoint_reshaping_and_interoperability.py:308

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/glm130b/checkpoint_reshaping_and_interoperability.py:271

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/starcoder/checkpoint_reshaping_and_interoperability.py:556

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/chatglm/checkpoint_reshaping_and_interoperability.py:298

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/yi/checkpoint_reshaping_and_interoperability.py:333

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/galactica/checkpoint_reshaping_and_interoperability.py:335

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.0.py:358

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(hgmodel, mgmodel, args)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_moe.py:205

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(hgmodel, mgmodel, args)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_mha_to_moe.py:53

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(hfmodel, mgmodel, args)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py:380

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(hgmodel, mgmodel, args)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_mha.py:158

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(args)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_gqa.py:250

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

(mgmodel, hgmodel, args, hf_config)

toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:398

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.5.py:381

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/bloom/checkpoint_reshaping_and_interoperability.py:541

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/bloom/reward_model_to_megatron.py:540

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_megatron

Convert a checkpoint from HuggingFace Transformers to Megatron-LM. This allows converted checkpoints with variable tensor parallelism and pip

toolkits/model_checkpoints_convertor/baichuan2/checkpoint_reshaping_and_interoperability.py:333

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_te

(args)

toolkits/model_checkpoints_convertor/baichuan/hf2te.py:212

↓ 1 callersFunctionconvert_checkpoint_from_transformers_to_te

(args)

toolkits/model_checkpoints_convertor/baichuan2/hf2te.py:196

↓ 1 callersFunctioncore_gpt_dataset_config_from_args

(args)

examples/linear_moe_mixtral/pretrain_mcore_mistral.py:252

↓ 1 callersFunctioncore_gpt_dataset_config_from_args

(args)

examples/linear_moe_deepseek_v2/pretrain_deepseek.py:249

↓ 1 callersFunctioncore_gpt_dataset_config_from_args

(args)

examples/linear_moe_qwen2/pretrain_qwen.py:240

↓ 1 callersFunctioncore_gpt_dataset_config_from_args

(args)

examples/linear_llama3/pretrain_llama.py:243

↓ 1 callersFunctioncreate_huggingface_model

(args)

toolkits/model_checkpoints_convertor/llama/hf2mcore.py:174

↓ 1 callersFunctioncreate_huggingface_model

(args)

toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:140

↓ 1 callersFunctioncreate_huggingface_model

(args)

toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:168

↓ 1 callersFunctioncreate_megatron_model

(args, hf_config)

toolkits/model_checkpoints_convertor/llama/hf2mcore.py:186

↓ 1 callersFunctioncreate_megatron_model

(args, hf_config)

toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:151

↓ 1 callersFunctioncreate_megatron_model

(args, hf_config)

toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:180

↓ 1 callersMethodcreate_model_inputs

(self, tokens)

linear_moe/lm_evaluate.py:83

↓ 1 callersFunctiondefault_cache_dir

()

linear_moe/sequence_modeling/mamba2/triton_cache_manager.py:19

↓ 1 callersMethoddefault_decoder_cross_attention

Cross attention for a standard encoder-decoder model.

linear_moe/model/llama3/transformer_legacy.py:946

↓ 1 callersMethoddraw_box

(self, box_coord, alpha=0.5, edge_color="g", line_style="-")

linear_moe/tokenizer/tokenization_qwen_vl.py:533

↓ 1 callersMethoddraw_text

( self, text, position, *, font_size=None, color="g",

linear_moe/tokenizer/tokenization_qwen_vl.py:500

↓ 1 callersMethodencode

(self, text)

toolkits/pretrain_data_preprocessing/preprocess_data.py:41

↓ 1 callersFunctionevaluate

Evaluation.

examples/linear_moe_qwen2/evaluate_huggingface_qwen.py:79

↓ 1 callersFunctionevaluate

Evaluation.

examples/linear_moe_qwen2/evaluate_mcore_qwen.py:145

↓ 1 callersFunctionevaluate

Evaluation.

linear_moe/training.py:701

↓ 1 callersFunctionfinish_mpu_init

()

linear_moe/initialize.py:53

↓ 1 callersFunctionforward_step

Forward step.

examples/linear_moe_qwen2/evaluate_huggingface_qwen.py:52

↓ 1 callersFunctionforward_step

Forward step.

examples/linear_moe_qwen2/evaluate_mcore_qwen.py:112

↓ 1 callersMethodgather_indices

Gather tensors and concatinate along the first dimension.

linear_moe/model/llama3/transformer_legacy.py:220

← previous next →401–500 of 1,498, ranked by callers