MCPcopy Create free account

hub / github.com/OpenSparseLLMs/Linear-MoE / functions

Functions1,498 in github.com/OpenSparseLLMs/Linear-MoE

⨍ Functions 1,498◇ Types & classes 227
↓ 3 callersFunctioncopy_huggingface_tokenizer
(src_path, dst_path, with_code=False)
toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:195
↓ 3 callersFunctionexpand2square
(pil_img, background_color)
linear_moe/data/llava/mm_utils.py:27
↓ 3 callersFunctionforward_step
Forward training step. Args: data_iterator : Input data iterator model (GPTModel): The GPT Model
examples/linear_llama3/pretrain_llama.py:218
↓ 3 callersMethodget_blank_token
(length: int)
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:158
↓ 3 callersMethodget_command
(self, token)
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:305
↓ 3 callersFunctionget_hybrid_lasp2_linear_moe_layer_local_spec
( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )
linear_moe/model/qwen2/layer_specs.py:753
↓ 3 callersFunctionget_hybrid_lightning_attention_linear_moe_layer_local_spec
( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )
linear_moe/model/qwen2/layer_specs.py:690
↓ 3 callersMethodget_tab_token
()
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:163
↓ 3 callersFunctionlayer_norm_fn
( x, weight, bias, residual=None, eps=1e-6, prenorm=False, residual_in_fp32=False,
linear_moe/model/common_modules/layernorm.py:469
↓ 3 callersFunctionlayer_norm_linear_fn
( x, norm_weight, norm_bias, linear_weight, linear_bias, residual=None, eps=1e-6,
linear_moe/model/common_modules/layernorm.py:774
↓ 3 callersFunctionmegatron_to_transformers_fix_query_key_value_ordering
Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :] for compatibility with later versions of NVIDIA Megatron-LM. The
toolkits/model_checkpoints_convertor/mistral/hf2megatron.py:217
↓ 3 callersFunctionname_to_expert_rank
(key)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py:60
↓ 3 callersMethodpreprocess
Preprocess an image or batch of images. Args: images (`ImageInput`): Image to preprocess. do
linear_moe/data/llava/cvcuda_image_processing_clip.py:236
↓ 3 callersFunctionsave_state_dict
(args, model, checkpoint_name)
toolkits/model_checkpoints_convertor/deepseek/hf2mcore_deepseek_v2_moe.py:263
↓ 3 callersFunctionsave_state_dict
(args, model, checkpoint_name)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_moe.py:253
↓ 3 callersFunctionsave_state_dict
(args, model, checkpoint_name)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_mha_to_moe.py:104
↓ 3 callersMethodset_layer_number
Set the layer number for the router.
linear_moe/model/qwen2/moe/router.py:175
↓ 3 callersMethodstep
(self, hidden_states, conv_state, ssm_state)
linear_moe/sequence_modeling/mamba2/mamba_mixer.py:351
↓ 3 callersFunctionswitch_load_balancing_loss_func
Calculate the auxiliary loss for better load balacing. Please refer to the Switch Transformer paper (https://arxiv.org/abs/2101.03961) for detail
linear_moe/model/mixtral/moe/moe_utils.py:17
↓ 3 callersFunctiontransformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/llama/hf2megatron.py:260
↓ 3 callersFunctiontransformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.5.py:279
↓ 3 callersFunctionyarn_get_mscale
(scale=1, mscale=1)
linear_moe/model/deepseek_v2/yarn_rotary_pos_embedding.py:83
↓ 2 callersMethod__init__
( self, input_dim: int, output_dim: int, low_rank_dim: int, bias: Opti
linear_moe/sequence_modeling/rwkv6/dd_lerp_linear.py:86
↓ 2 callersMethod__init__
( self, config: TransformerConfig, submodules: Union[SelfAttentionSubmodules, CrossAtt
linear_moe/model/mixtral/transformer/attention.py:61
↓ 2 callersMethod__init__
( self, config: TransformerConfig, submodules: Union[SelfAttentionSubmodules, CrossAtt
linear_moe/model/qwen2/transformer/attention.py:61
↓ 2 callersMethod__init__
Initialize the MoE Token Dispatcher.
linear_moe/model/qwen2/moe/token_dispatcher.py:31
↓ 2 callersMethod__init__
Initialize the MoE Token Dispatcher.
linear_moe/model/deepseek_v2/moe/token_dispatcher.py:30
↓ 2 callersMethod__init__
(self, hidden_size, init_method)
linear_moe/model/llama3/language_model.py:95
↓ 2 callersMethod__init__
( self, config: TransformerConfig, submodules: Union[SelfAttentionSubmodules, CrossAtt
linear_moe/model/llama3/transformer/attention.py:65
↓ 2 callersMethod_adjust_key_value_for_inference
Saves the generated key and value tensors to the end of the buffers in inference_params. Returns the full size keys and values from t
linear_moe/sequence_modeling/mom_linear_attention.py:435
↓ 2 callersMethod_allocate_memory
Allocate memory to store kv cache during inference.
linear_moe/sequence_modeling/mom_linear_attention.py:423
↓ 2 callersMethod_allocate_memory
Allocate memory to store kv cache during inference.
linear_moe/sequence_modeling/linear_attention.py:260
↓ 2 callersMethod_allocate_memory
Allocate memory to store kv cache during inference.
linear_moe/model/mixtral/transformer/attention.py:158
↓ 2 callersMethod_allocate_memory
Allocate memory to store kv cache during inference.
linear_moe/model/qwen2/transformer/attention.py:158
↓ 2 callersMethod_allocate_memory
(self, inference_max_sequence_len, batch_size, num_attention_heads)
linear_moe/model/llama3/transformer_legacy.py:639
↓ 2 callersMethod_allocate_memory
Allocate memory to store kv cache during inference.
linear_moe/model/llama3/transformer/attention.py:162
↓ 2 callersFunction_build_train_valid_test_datasets
(data_prefix, max_padding_length, dataset_type, splits_string, train_vali
linear_moe/data/__init__.py:187
↓ 2 callersMethod_compute_inv_freq
(self, device=None)
linear_moe/model/common_modules/rotary.py:201
↓ 2 callersMethod_get_layer
(self, layer_number)
linear_moe/model/llama3/transformer_legacy.py:1608
↓ 2 callersMethod_get_layer_offset
(self)
linear_moe/model/llama3/transformer_layer.py:153
↓ 2 callersFunction_layer_norm_bwd
( dy, x, weight, bias, eps, mean, rstd, dresidual=None, has_residual=False
linear_moe/model/common_modules/layernorm.py:303
↓ 2 callersFunction_layer_norm_fwd
( x, weight, bias, eps, residual=None, out_dtype=None, residual_dtype=None, is
linear_moe/model/common_modules/layernorm.py:134
↓ 2 callersFunction_list_find
( input_list: List[Any], candidates: Tuple[Any], start: int = 0, )
linear_moe/tokenizer/tokenization_qwen_vl.py:53
↓ 2 callersMethod_preprocess
(self, text: str, linebreak=True, whitespaces=True)
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:195
↓ 2 callersMethod_set_cos_sin_cache
(self, seq_len, device, dtype)
linear_moe/model/deepseek_v2/yarn_rotary_pos_embedding.py:39
↓ 2 callersMethod_tokenize
Returns a tokenized string.
linear_moe/tokenizer/tokenization_yi.py:106
↓ 2 callersFunction_tokenize_fn
Tokenize a list of strings.
linear_moe/data/llava/mm_pretrain_dataset.py:35
↓ 2 callersMethod_update_cos_sin_cache
(self, seqlen, device=None, dtype=None)
linear_moe/model/common_modules/rotary.py:207
↓ 2 callersFunctionbuild_evaluation_dataset
(dataset)
linear_moe/data/__init__.py:29
↓ 2 callersFunctionbuild_huggingface_model
(model_to_load, compute_dtype, random_init=False)
toolkits/model_checkpoints_convertor/llama/hf2mcore.py:129
↓ 2 callersFunctionbuild_huggingface_model
(model_to_load, compute_dtype, random_init=False)
toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:117
↓ 2 callersFunctionbuild_huggingface_model
(model_to_load, compute_dtype, random_init=False)
toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:122
↓ 2 callersFunctioncheck_layer
(layers_to_copy, k)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py:454
↓ 2 callersFunctioncompute_weight_and_optimizer_memory
(args, verbose=False)
linear_moe/utils.py:22
↓ 2 callersMethodconvert_tokens_to_ids
( self, tokens: Union[bytes, str, List[Union[bytes, str]]] )
linear_moe/tokenizer/tokenization_qwen_vl.py:176
↓ 2 callersMethoddecode
(self, text_ids: List[int], special_tokens=False)
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:218
↓ 2 callersFunctiondetokenize_generations
Detokenize the generated tokens. Args: tokens_gpu_tensor (torch.Tensor): The generated tokens as a GPU tensor. lengths_gpu_t
linear_moe/generation/tokenization.py:24
↓ 2 callersFunctionelu_p1
(x)
linear_moe/sequence_modeling/deltanet/deltanet.py:19
↓ 2 callersMethodencode
(self, text: str, linebreak=True, whitespaces=True,
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:203
↓ 2 callersFunctionflatten_diag_outer_product
(x, y)
linear_moe/model/common_modules/feature_map.py:18
↓ 2 callersFunctionflatten_diag_outer_product_off1
(x, y)
linear_moe/model/common_modules/feature_map.py:26
↓ 2 callersMethodgather_indices
Gather tensors and concatenate along the first dimension. Args: local_indices (torch.Tensor): Tensor of indices
linear_moe/model/mixtral/moe/token_dispatcher.py:96
↓ 2 callersFunctiongenerate_and_post_process
Run inference and post-process outputs, i.e., detokenize, move to cpu and convert to list. Args: model (torch.nn.Module): The mo
linear_moe/generation/api.py:25
↓ 2 callersFunctionget_checkpoint_sub_dir_name
(tp_rank, pp_rank, pp_size, ep_rank, ep_size)
toolkits/model_checkpoints_convertor/mistral/hf2mcore_mixtral.py:214
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/falcon40b/checkpoint_reshaping_and_interoperability.py:263
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/llama/hf2mcore_70b.py:151
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/llama/hf2megatron.py:323
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/mistral/hf2megatron.py:313
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/mistral/hf2mcore_mixtral.py:221
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/baichuan/checkpoint_reshaping_and_interoperability.py:307
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/falcon/checkpoint_reshaping_and_interoperability.py:265
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/starcoder/checkpoint_reshaping_and_interoperability.py:287
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.0.py:319
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_gqa.py:130
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.5.py:342
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/bloom/checkpoint_reshaping_and_interoperability.py:265
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/bloom/reward_model_to_megatron.py:265
↓ 2 callersFunctionget_megatron_sharded_states
Get sharded checkpoints from NVIDIA Megatron-LM checkpoint based on the provided tensor parallel size, pipeline parallel size and pipeline pa
toolkits/model_checkpoints_convertor/baichuan2/checkpoint_reshaping_and_interoperability.py:294
↓ 2 callersFunctiongrouped_gemm_is_available
()
linear_moe/model/mixtral/moe/grouped_gemm_util.py:20
↓ 2 callersFunctionloss_func
Loss function. Args: loss_mask (torch.Tensor): Used to mask out some portions of the loss output_tensor (torch.Tensor): The tenso
examples/linear_llama3/pretrain_llama.py:186
↓ 2 callersFunctionmain
Main program.
examples/linear_moe_qwen2/evaluate_mcore_qwen.py:171
↓ 2 callersFunctionmegatron_to_transformers_fix_query_key_value_ordering
Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :] for compatibility with later versions of NVIDIA Megatron-LM. The
toolkits/model_checkpoints_convertor/llama/hf2mcore_70b.py:171
↓ 2 callersFunctionmegatron_to_transformers_fix_query_key_value_ordering
Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :] for compatibility with later versions of NVIDIA Megatron-LM. The
toolkits/model_checkpoints_convertor/mistral/hf2mcore_mixtral.py:253
↓ 2 callersFunctionmegatron_to_transformers_fix_query_key_value_ordering
Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :] for compatibility with later versions of NVIDIA Megatron-LM. The
toolkits/model_checkpoints_convertor/baichuan/checkpoint_reshaping_and_interoperability.py:211
↓ 2 callersFunctionmegatron_to_transformers_fix_query_key_value_ordering
Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :] for compatibility with later versions of NVIDIA Megatron-LM. The
toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.0.py:222
↓ 2 callersFunctionmegatron_to_transformers_fix_query_key_value_ordering
Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :] for compatibility with later versions of NVIDIA Megatron-LM. The
toolkits/model_checkpoints_convertor/baichuan2/checkpoint_reshaping_and_interoperability.py:213
↓ 2 callersFunctionmodel_provider
Builds the model. If you set the use_mcore_models to True, it will return the mcore GPT model and if not the legacy GPT model. Args:
examples/linear_moe_deepseek_v2/pretrain_deepseek.py:57
↓ 2 callersFunctionmodel_provider
(pre_process=True, post_process=True)
examples/linear_moe_qwen2/pretrain_qwen.py:57
↓ 2 callersFunctionname_to_expert_rank
(key)
toolkits/model_checkpoints_convertor/deepseek/hf2mcore_deepseek_v2_moe.py:52
↓ 2 callersFunctionname_to_expert_rank
(key)
toolkits/model_checkpoints_convertor/llama/hf2mcore.py:215
↓ 2 callersFunctionname_to_expert_rank
(key)
toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:173
↓ 2 callersFunctionname_to_expert_rank
(key)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_moe.py:51
↓ 2 callersFunctionname_to_expert_rank
(key)
toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:208
↓ 2 callersMethodprint_processing_stats
(self, count, proc_start, total_bytes_processed)
toolkits/pretrain_data_preprocessing/preprocess_data_megatron.py:102
↓ 2 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/falcon40b/checkpoint_reshaping_and_interoperability.py:189
↓ 2 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/llama/hf2mcore_70b.py:237
↓ 2 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/llama/hf2megatron.py:200
↓ 2 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/mistral/hf2megatron.py:190
↓ 2 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/baichuan/checkpoint_reshaping_and_interoperability.py:184
↓ 2 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/falcon/checkpoint_reshaping_and_interoperability.py:191
← previousnext →101–200 of 1,498, ranked by callers