MCPcopy Create free account

hub / github.com/OpenSparseLLMs/Linear-MoE / functions

Functions1,498 in github.com/OpenSparseLLMs/Linear-MoE

⨍ Functions 1,498◇ Types & classes 227
↓ 1 callersFunctionmerge_transformers_sharded_states_13b
Merge sharded checkpoints from transformers into a single checkpoint. Args: path (str): the path to the sharded checkpoints n
toolkits/model_checkpoints_convertor/baichuan2/checkpoint_reshaping_and_interoperability.py:279
↓ 1 callersFunctionmerge_transformers_sharded_states_7b
Merge sharded checkpoints from transformers into a single checkpoint. Args: path (str): the path to the sharded checkpoints n
toolkits/model_checkpoints_convertor/falcon40b/checkpoint_reshaping_and_interoperability.py:248
↓ 1 callersFunctionmerge_transformers_sharded_states_7b
Merge sharded checkpoints from transformers into a single checkpoint. Args: path (str): the path to the sharded checkpoints n
toolkits/model_checkpoints_convertor/llama/hf2megatron.py:293
↓ 1 callersFunctionmerge_transformers_sharded_states_7b
Merge sharded checkpoints from transformers into a single checkpoint. Args: path (str): the path to the sharded checkpoints n
toolkits/model_checkpoints_convertor/mistral/hf2megatron.py:283
↓ 1 callersFunctionmerge_transformers_sharded_states_7b
Merge sharded checkpoints from transformers into a single checkpoint. Args: path (str): the path to the sharded checkpoints n
toolkits/model_checkpoints_convertor/falcon/checkpoint_reshaping_and_interoperability.py:250
↓ 1 callersFunctionmerge_transformers_sharded_states_7b
Merge sharded checkpoints from transformers into a single checkpoint. Args: path (str): the path to the sharded checkpoints n
toolkits/model_checkpoints_convertor/yi/checkpoint_reshaping_and_interoperability.py:264
↓ 1 callersFunctionmerge_transformers_sharded_states_7b
Merge sharded checkpoints from transformers into a single checkpoint. Args: path (str): the path to the sharded checkpoints n
toolkits/model_checkpoints_convertor/galactica/checkpoint_reshaping_and_interoperability.py:266
↓ 1 callersFunctionmodel_provider
(pre_process=True, post_process=True)
examples/linear_moe_mixtral/pretrain_mcore_mistral.py:71
↓ 1 callersFunctionparallel_lm_logits
LM logits using word embedding weights.
linear_moe/model/llama3/language_model.py:19
↓ 1 callersFunctionparse_arguments
()
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron_ori.py:22
↓ 1 callersFunctionparse_arguments
()
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron.py:23
↓ 1 callersFunctionpost_language_model_processing
(lm_output, labels, logit_weights, parallel_output,
linear_moe/model/llama3/gpt_model.py:28
↓ 1 callersFunctionprepare_dm
(q, k, v, do, BLOCK=128, CBLOCK=64)
linear_moe/sequence_modeling/lasp2/lasp2_with_mask_triton_op.py:823
↓ 1 callersFunctionprepare_dm
(q, k, v, do, BLOCK=128, CBLOCK=64)
linear_moe/sequence_modeling/lasp2/lasp2_without_mask_triton_op.py:564
↓ 1 callersFunctionprepare_m
(q, k, v, BLOCK=128, CBLOCK=64)
linear_moe/sequence_modeling/lasp2/lasp2_with_mask_triton_op.py:715
↓ 1 callersFunctionprepare_m
(q, k, v, BLOCK=128, CBLOCK=64)
linear_moe/sequence_modeling/lasp2/lasp2_without_mask_triton_op.py:471
↓ 1 callersFunctionpreprocess
Preprocess conversation data for the model input. Parameters: sources (List[Dict]): A list of conversation segments. tokeniz
linear_moe/data/qwen_vl.py:30
↓ 1 callersFunctionpreprocess
Given a list of sources, each is a conversation list. This transform: 1. Add signal '### ' at the beginning each sentence, with end signal '\
linear_moe/data/llava/mm_pretrain_dataset.py:127
↓ 1 callersMethodpreprocess
Preprocess token indices for AlltoAll communication and token permutation. This method computes the number of tokens assigned to each expert
linear_moe/model/qwen2/moe/token_dispatcher.py:337
↓ 1 callersMethodpreprocess
Preprocess token indices for AlltoAll communication and token permutation. This method computes the number of tokens assigned to each expert
linear_moe/model/deepseek_v2/moe/token_dispatcher.py:320
↓ 1 callersMethodpreprocess
Preprocess the data by tokenizing.
linear_moe/data/starcoder.py:87
↓ 1 callersFunctionpreprocess_multimodal
(sources: Sequence[str])
linear_moe/data/llava/mm_pretrain_dataset.py:90
↓ 1 callersFunctionpreprocess_plain
( sources: Sequence[str], tokenizer: transformers.PreTrainedTokenizer, )
linear_moe/data/llava/mm_pretrain_dataset.py:107
↓ 1 callersFunctionprocess_batch
Process batch and produce inputs for the model.
linear_moe/finetune_utils.py:36
↓ 1 callersMethodprocess_samples_from_single_path
Process a single file containing prompt-answer pairs and return a list of samples.
linear_moe/data/bloom.py:65
↓ 1 callersMethodprocess_samples_from_single_path
(self, filename)
linear_moe/data/glm.py:150
↓ 1 callersMethodprocess_samples_from_single_path
(self, filename)
linear_moe/data/glm.py:266
↓ 1 callersMethodprocess_single_file
Processes a single dataset file. Args: path(str): The path of the dataset file.
linear_moe/data/glm.py:50
↓ 1 callersFunctionreconstruct
(re_x, indices: torch.Tensor, sorted_indices: torch.Tensor, batch_size: int, seq_len: int, topk: int, routing_
linear_moe/sequence_modeling/mom_linear_attention.py:82
↓ 1 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/mistral/hf2mcore_mixtral.py:319
↓ 1 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/glm/checkpoint_reshaping_and_interoperability.py:128
↓ 1 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/glm130b/checkpoint_reshaping_and_interoperability.py:128
↓ 1 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/chatglm/checkpoint_reshaping_and_interoperability.py:156
↓ 1 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/yi/checkpoint_reshaping_and_interoperability.py:171
↓ 1 callersFunctionrecursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/galactica/checkpoint_reshaping_and_interoperability.py:173
↓ 1 callersFunctionrepetition_penalty
Implement the repetition penalty, check paper https://arxiv.org/pdf/1909.05858.pdf
linear_moe/generation/generation.py:109
↓ 1 callersFunctionreplace_mlp_with_moe
(args, model)
toolkits/model_checkpoints_convertor/llama/hf2mcore.py:151
↓ 1 callersFunctionreplace_mlp_with_moe
(args, model)
toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:144
↓ 1 callersMethodrescale
Rescale an image by a scale factor. image = image * scale. Args: image (`np.ndarray`): Image to rescale.
linear_moe/data/llava/cvcuda_image_processing_clip.py:186
↓ 1 callersMethodreset_image
(self, img)
linear_moe/tokenizer/tokenization_qwen_vl.py:471
↓ 1 callersMethodretro_decoder_cross_attention
Cross attention for Retro decoder. Notation: ns : Sequence length. bs : Batch size. d : Hidden size.
linear_moe/model/llama3/transformer_legacy.py:1049
↓ 1 callersMethodretro_encoder_cross_attention
Cross attention for Retro encoder. Notation: ns : Sequence length. bs : Batch size. d : Hidden size.
linear_moe/model/llama3/transformer_legacy.py:982
↓ 1 callersFunctionrms_norm_fn
( x, weight, bias, residual=None, eps=1e-6, prenorm=False, residual_in_fp32=False
linear_moe/model/common_modules/layernorm.py:515
↓ 1 callersFunctionrotate_half
(x, interleaved=False)
linear_moe/model/common_modules/rotary.py:13
↓ 1 callersMethodrouting
Routing function. Args: logits (torch.Tensor): Logits tensor. Returns: Tuple[torch.Tensor, torch.Tensor]: Tu
linear_moe/model/mixtral/moe/router.py:69
↓ 1 callersMethodrouting
Top-k routing function Args: logits (torch.Tensor): Logits tensor after gating. Returns: probs (torch.Tensor
linear_moe/model/qwen2/moe/router.py:324
↓ 1 callersMethodrouting
Top-k routing function Args: logits (torch.Tensor): Logits tensor. Returns: Tuple[torch.Tensor, torch.Tensor
linear_moe/model/deepseek_v2/moe/router_old.py:238
↓ 1 callersMethodrouting
Top-k routing function Args: logits (torch.Tensor): Logits tensor after gating. Returns: probs (torch.Tensor
linear_moe/model/deepseek_v2/moe/router.py:324
↓ 1 callersMethodrun_realtime_tests
Performs a consistency check. This function makes sure that tensors across devices are the same during an experiment. This is often n
linear_moe/model/mixtral/transformer/attention.py:400
↓ 1 callersMethodrun_realtime_tests
Performs a consistency check. This function makes sure that tensors across devices are the same during an experiment. This is often n
linear_moe/model/qwen2/transformer/attention.py:400
↓ 1 callersMethodrun_realtime_tests
Performs a consistency check. This function makes sure that tensors across devices are the same during an experiment. This is often n
linear_moe/model/llama3/transformer/attention.py:398
↓ 1 callersFunctionsave_hfmodel
(args, model)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py:741
↓ 1 callersFunctionsave_hgmodel
(args, model)
toolkits/model_checkpoints_convertor/deepseek/hf2mcore_deepseek_v2_moe.py:394
↓ 1 callersFunctionsave_hgmodel
(args, model)
toolkits/model_checkpoints_convertor/llama/hf2mcore.py:571
↓ 1 callersFunctionsave_hgmodel
(args, model)
toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:497
↓ 1 callersFunctionsave_hgmodel
(args, model)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_moe.py:375
↓ 1 callersFunctionsave_hgmodel
(args, model)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_mha.py:266
↓ 1 callersFunctionsave_hgmodel
(args, model)
toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:580
↓ 1 callersFunctionsave_mgmodel
(mgmodel, args)
toolkits/model_checkpoints_convertor/deepseek/hf2mcore_deepseek_v2_moe.py:274
↓ 1 callersFunctionsave_mgmodel
(args, mgmodel, load_path, save_path)
toolkits/model_checkpoints_convertor/llama/hf2mcore.py:447
↓ 1 callersFunctionsave_mgmodel
(args, mgmodel, load_path, save_path)
toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:373
↓ 1 callersFunctionsave_mgmodel
(mgmodel, args)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_moe.py:264
↓ 1 callersFunctionsave_mgmodel
(mgmodel, args)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_mha_to_moe.py:115
↓ 1 callersFunctionsave_mgmodel
(mgmodel, args)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py:459
↓ 1 callersFunctionsave_mgmodel
(mgmodel, args)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_mha.py:201
↓ 1 callersFunctionsave_mgmodel
(args, mgmodel, load_path, save_path)
toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:467
↓ 1 callersFunctionscore_and_return_on_first_stage
Function for just scoring. Args: model: no interleaving is supported. tokens: prompt tokens extended to be of size [b, max_p
linear_moe/generation/generation.py:31
↓ 1 callersMethodset_layer_number
Set the layer number for the router.
linear_moe/model/deepseek_v2/moe/router.py:175
↓ 1 callersMethodsharded_state_dict
( self, prefix: str = '', sharded_offsets: tuple = (), metadata: Optional[dict] = None )
linear_moe/model/mixtral/transformer/mlp.py:157
↓ 1 callersMethodsharded_state_dict
( self, prefix: str = '', sharded_offsets: tuple = (), metadata: Optional[dict] = None )
linear_moe/model/qwen2/transformer/mlp.py:157
↓ 1 callersMethodsharded_state_dict
(self, prefix='', sharded_offsets=(), metadata=None)
linear_moe/model/qwen2/moe/experts.py:184
↓ 1 callersMethodsharded_state_dict
( self, prefix: str = '', sharded_offsets: tuple = (), metadata: Optional[dict] = None )
linear_moe/model/deepseek_v2/transformer/mlp.py:145
↓ 1 callersMethodsharded_state_dict
(self, prefix='', sharded_offsets=(), metadata=None)
linear_moe/model/deepseek_v2/moe/experts.py:173
↓ 1 callersMethodsharded_state_dict
( self, prefix: str = '', sharded_offsets: tuple = (), metadata: Optional[dict] = None )
linear_moe/model/llama3/transformer/mlp.py:138
↓ 1 callersMethodshared_forward
( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None,
linear_moe/sequence_modeling/mom_linear_attention.py:507
↓ 1 callersFunctionsinkhorn
(cost, tol=0.0001)
linear_moe/model/llama3/transformer_legacy.py:174
↓ 1 callersMethodsinkhorn_load_balancing
Apply sinkhorn routing to the logits tensor. Args: logits (torch.Tensor): The logits tensor. Returns: torch.
linear_moe/model/mixtral/moe/router.py:119
↓ 1 callersMethodsinkhorn_load_balancing
Apply sinkhorn routing to the logits tensor. Args: logits (torch.Tensor): The logits tensor. Returns: torch.
linear_moe/model/qwen2/moe/router.py:194
↓ 1 callersMethodsinkhorn_load_balancing
Apply sinkhorn routing to the logits tensor. Args: logits (torch.Tensor): The logits tensor. Returns: torch.
linear_moe/model/deepseek_v2/moe/router_old.py:120
↓ 1 callersMethodsinkhorn_load_balancing
Apply sinkhorn routing to the logits tensor. Args: logits (torch.Tensor): The logits tensor. Returns: torch.
linear_moe/model/deepseek_v2/moe/router.py:194
↓ 1 callersFunctionsqrelu_bwd
(g, x)
linear_moe/model/common_modules/activations.py:284
↓ 1 callersFunctionsqrelu_fwd
(x)
linear_moe/model/common_modules/activations.py:278
↓ 1 callersMethodtoken_permutation
Dispatch tokens to experts. Args: tokens (torch.Tensor): Input tokens. indices (torch.Tensor): indices tensor.
linear_moe/model/mixtral/moe/token_dispatcher.py:36
↓ 1 callersMethodtoken_permutation
Dispatch tokens to experts. Args: tokens (torch.Tensor): Input tokens. indices (torch.Tensor): indices tensor.
linear_moe/model/qwen2/moe/token_dispatcher.py:38
↓ 1 callersMethodtoken_permutation
Dispatch tokens to experts. Args: tokens (torch.Tensor): Input tokens. indices (torch.Tensor): indices tensor.
linear_moe/model/deepseek_v2/moe/token_dispatcher.py:37
↓ 1 callersMethodtoken_unpermutation
Restores the expert output to its original ordering. Args: expert_output (torch.Tensor): The output tensor from the expert models
linear_moe/model/mixtral/moe/token_dispatcher.py:51
↓ 1 callersMethodtoken_unpermutation
Restores the expert output to its original ordering. Args: expert_output (torch.Tensor): The output tensor from the expert models
linear_moe/model/qwen2/moe/token_dispatcher.py:53
↓ 1 callersMethodtoken_unpermutation
Restores the expert output to its original ordering. Args: expert_output (torch.Tensor): The output tensor from the expert models
linear_moe/model/deepseek_v2/moe/token_dispatcher.py:52
↓ 1 callersMethodtokenize
(self, *text)
toolkits/pretrain_data_preprocessing/preprocess_data_megatron.py:28
↓ 1 callersMethodtokenize
This function tokenizes the source strings given the tokenizer and returns a dictionary containing the tokenized inputs and labels.
linear_moe/data/starcoder.py:101
↓ 1 callersMethodtokenize
(self, text)
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:341
↓ 1 callersFunctiontrain
Train the model function.
linear_moe/training.py:566
↓ 1 callersFunctiontransform
transform the hidden_states into chunks by experts (expert_batch, selected_len, hidden_size) expert_batch may be close to experts * orginal_b
linear_moe/sequence_modeling/mom_linear_attention.py:15
↓ 1 callersFunctiontransformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/falcon40b/checkpoint_reshaping_and_interoperability.py:215
↓ 1 callersFunctiontransformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/baichuan/hf2te.py:132
↓ 1 callersFunctiontransformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/baichuan/checkpoint_reshaping_and_interoperability.py:244
↓ 1 callersFunctiontransformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/baichuan2/hf2te.py:131
↓ 1 callersFunctiontransformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/baichuan2/checkpoint_reshaping_and_interoperability.py:246
↓ 1 callersMethodtruncate
Truncates an array to a maximum length or pads it with zeros if its length is less than `max_length`. Args: tokenizer: Th
linear_moe/data/bloom.py:48
↓ 1 callersFunctionyarn_find_correction_range
( low_rot, high_rot, dim, base=10000, max_position_embeddings=2048 )
linear_moe/model/deepseek_v2/yarn_rotary_pos_embedding.py:71
← previousnext →601–700 of 1,498, ranked by callers