Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/OpenSparseLLMs/Linear-MoE
/ functions
Functions
1,498 in github.com/OpenSparseLLMs/Linear-MoE
⨍
Functions
1,498
◇
Types & classes
227
↓ 2 callers
Function
recursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/starcoder/checkpoint_reshaping_and_interoperability.py:179
↓ 2 callers
Function
recursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.0.py:195
↓ 2 callers
Function
recursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.5.py:219
↓ 2 callers
Function
recursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/bloom/checkpoint_reshaping_and_interoperability.py:157
↓ 2 callers
Function
recursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/bloom/reward_model_to_megatron.py:157
↓ 2 callers
Function
recursive_print
Recursively print the structure of a checkpoint. This function is taken from `convert_megatron_gpt2_checkpoint.py` Args: name (str):
toolkits/model_checkpoints_convertor/baichuan2/checkpoint_reshaping_and_interoperability.py:186
↓ 2 callers
Function
rotate_half
Rotates half the hidden dims of the input.
linear_moe/model/deepseek_v2/yarn_rotary_pos_embedding.py:166
↓ 2 callers
Function
save_state_dict
(args, model, checkpoint_name)
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_mha.py:190
↓ 2 callers
Method
set_input_tensor
Sets input tensor to the model. See megatron.model.transformer.set_input_tensor() Args: input_tensor (Tensor): Sets the
linear_moe/model/qwen2/model.py:143
↓ 2 callers
Function
sum_norm
(x)
linear_moe/sequence_modeling/gated_deltanet/gated_deltanet.py:23
↓ 2 callers
Function
sum_norm
(x)
linear_moe/sequence_modeling/deltanet/deltanet.py:24
↓ 2 callers
Method
to_list_format
(self, text: str)
linear_moe/tokenizer/tokenization_qwen_vl.py:336
↓ 2 callers
Function
tokenize_prompts
Tokenize prompts and make them avaiable on all ranks. Args: prompts (list): List of prompts to be tokenized. tokens_to_gener
linear_moe/generation/tokenization.py:70
↓ 2 callers
Function
topk_softmax_with_capacity
Apply capacity and padding to the top-k selection. Args: logits (torch.Tensor): Logits tensor. topk (int): The number
linear_moe/model/qwen2/moe/router.py:41
↓ 2 callers
Function
topk_softmax_with_capacity
Apply capacity and padding to the top-k selection. Args: logits (torch.Tensor): Logits tensor. topk (int): The number
linear_moe/model/deepseek_v2/moe/router.py:41
↓ 2 callers
Function
train_step
Single training step.
linear_moe/training.py:307
↓ 2 callers
Function
training_log
Log training information such as losses, timing, ....
linear_moe/training.py:382
↓ 2 callers
Function
transformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/llama/hf2mcore_70b.py:204
↓ 2 callers
Function
transformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/mistral/hf2megatron.py:250
↓ 2 callers
Function
transformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/mistral/hf2mcore_mixtral.py:286
↓ 2 callers
Function
transformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/yi/checkpoint_reshaping_and_interoperability.py:231
↓ 2 callers
Function
transformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/galactica/checkpoint_reshaping_and_interoperability.py:233
↓ 2 callers
Function
transformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.0.py:255
↓ 2 callers
Function
transformers_to_megatron_fix_query_key_value_ordering
Permutes layout of param tensor to the one compatible with respective NVIDIA Megatron-LM chekpoint versions. Input is [num_splits * num_heads
toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_gqa.py:183
↓ 2 callers
Function
yarn_find_correction_dim
( num_rotations, dim, base=10000, max_position_embeddings=2048 )
linear_moe/model/deepseek_v2/yarn_rotary_pos_embedding.py:62
↓ 1 callers
Method
IdToToken
(self, idx)
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:311
↓ 1 callers
Method
__init__
(self, *args, **kwargs)
toolkits/model_checkpoints_convertor/llama/hf_llama_moe/llama_moe.py:20
↓ 1 callers
Method
__init__
( self, vocab_size=250880, hidden_size=64, n_layer=2, n_head=8,
toolkits/model_checkpoints_convertor/falcon/configuration_RW.py:31
↓ 1 callers
Method
__init__
Initialize the MoE Token Dispatcher.
linear_moe/model/mixtral/moe/token_dispatcher.py:29
↓ 1 callers
Method
__init__
(self, num_local_experts: int, config: TransformerConfig)
linear_moe/model/mixtral/moe/experts.py:38
↓ 1 callers
Method
__init__
(self, config: TransformerConfig, submodules: MLPSubmodules = None)
linear_moe/model/mixtral/moe/moe_layer.py:62
↓ 1 callers
Method
__init__
Initialize the Router module. Args: config (TransformerConfig): Configuration object for the Transformer model.
linear_moe/model/mixtral/moe/router.py:36
↓ 1 callers
Method
__init__
(self, num_local_experts: int, config: TransformerConfig)
linear_moe/model/qwen2/moe/experts.py:45
↓ 1 callers
Method
__init__
( self, config: TransformerConfig, submodules: MLPSubmodules = None, layer_number: int = None )
linear_moe/model/qwen2/moe/moe_layer.py:85
↓ 1 callers
Method
__init__
Initialize the Router module. Args: config (TransformerConfig): Configuration object for the Transformer model.
linear_moe/model/qwen2/moe/router.py:120
↓ 1 callers
Method
__init__
(self, dim, max_position_embeddings=2048, base=10000, device=None)
linear_moe/model/deepseek_v2/yarn_rotary_pos_embedding.py:20
↓ 1 callers
Method
__init__
( self, config: TransformerConfig, submodules: Union[SelfAttentionSubmodules],
linear_moe/model/deepseek_v2/transformer/attention.py:49
↓ 1 callers
Method
__init__
Initialize the Router module. Args: config (TransformerConfig): Configuration object for the Transformer model.
linear_moe/model/deepseek_v2/moe/router_old.py:45
↓ 1 callers
Method
__init__
(self, num_local_experts: int, config: TransformerConfig)
linear_moe/model/deepseek_v2/moe/experts.py:38
↓ 1 callers
Method
__init__
( self, config: TransformerConfig, submodules: MLPSubmodules = None, layer_number: int = None )
linear_moe/model/deepseek_v2/moe/moe_layer.py:73
↓ 1 callers
Method
__init__
Initialize the Router module. Args: config (TransformerConfig): Configuration object for the Transformer model.
linear_moe/model/deepseek_v2/moe/router.py:120
↓ 1 callers
Method
__init__
(self, img, scale=1.0)
linear_moe/tokenizer/tokenization_qwen_vl.py:449
↓ 1 callers
Method
__init__
(self, max_blank_len=80)
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:262
↓ 1 callers
Function
_add_speaker_and_signal
Add speaker and start/end signal on each round.
linear_moe/data/llava/mm_pretrain_dataset.py:70
↓ 1 callers
Method
_adjust_key_value_for_inference
Saves the generated key and value tensors to the end of the buffers in inference_params. Returns the full size keys and values from t
linear_moe/sequence_modeling/linear_attention.py:272
↓ 1 callers
Method
_adjust_key_value_for_inference
Saves the generated key and value tensors to the end of the buffers in inference_params. Returns the full size keys and values from t
linear_moe/model/mixtral/transformer/attention.py:170
↓ 1 callers
Method
_adjust_key_value_for_inference
Saves the generated key and value tensors to the end of the buffers in inference_params. Returns the full size keys and values from t
linear_moe/model/qwen2/transformer/attention.py:170
↓ 1 callers
Method
_adjust_key_value_for_inference
Saves the generated key and value tensors to the end of the buffers in inference_params. Returns the full size keys and values from t
linear_moe/model/llama3/transformer/attention.py:174
↓ 1 callers
Function
_allocate_auto
( total_layers_count: int, target_attention_ratio: float, target_mlp_ratio: float )
linear_moe/sequence_modeling/mamba2/mamba_hybrid_layer_allocation.py:24
↓ 1 callers
Function
_allocate_override
(total_layers_count: int, override_pattern: str)
linear_moe/sequence_modeling/mamba2/mamba_hybrid_layer_allocation.py:61
↓ 1 callers
Function
_build_infinite_size_dataloader
Build a looped dataloader with infinite size.
linear_moe/finetune_utils.py:108
↓ 1 callers
Method
_build_layers
(self)
linear_moe/model/mixtral/transformer_block.py:157
↓ 1 callers
Method
_build_layers
(self)
linear_moe/model/mixtral/hybrid/hybrid_transformer_block.py:144
↓ 1 callers
Method
_build_layers
(self)
linear_moe/model/qwen2/transformer_block.py:157
↓ 1 callers
Method
_build_layers
(self)
linear_moe/model/qwen2/hybrid/hybrid_transformer_block.py:144
↓ 1 callers
Method
_build_layers
(self)
linear_moe/model/deepseek_v2/transformer_block.py:153
↓ 1 callers
Method
_build_layers
(self)
linear_moe/model/deepseek_v2/hybrid/hybrid_transformer_block.py:135
↓ 1 callers
Method
_build_layers
(self)
linear_moe/model/llama3/hybrid/hybrid_transformer_block.py:144
↓ 1 callers
Method
_build_slope_tensor
(n_attention_heads: int)
linear_moe/sequence_modeling/lightning_attention/lightning_attention.py:80
↓ 1 callers
Function
_build_train_valid_dataloaders
Traing and validation dataloaders.
linear_moe/finetune_utils.py:119
↓ 1 callers
Method
_checkpointed_attention_forward
Forward method with selective activation checkpointing.
linear_moe/model/mixtral/transformer/attention.py:113
↓ 1 callers
Method
_checkpointed_attention_forward
Forward method with selective activation checkpointing.
linear_moe/model/qwen2/transformer/attention.py:113
↓ 1 callers
Method
_checkpointed_attention_forward
Forward method with activation checkpointing.
linear_moe/model/llama3/transformer_legacy.py:616
↓ 1 callers
Method
_checkpointed_attention_forward
Forward method with selective activation checkpointing.
linear_moe/model/llama3/transformer/attention.py:117
↓ 1 callers
Method
_checkpointed_forward
Forward method with activation checkpointing.
linear_moe/model/mixtral/transformer_block.py:209
↓ 1 callers
Method
_checkpointed_forward
Forward method with activation checkpointing.
linear_moe/model/mixtral/hybrid/hybrid_transformer_block.py:201
↓ 1 callers
Method
_checkpointed_forward
Forward method with activation checkpointing.
linear_moe/model/qwen2/transformer_block.py:209
↓ 1 callers
Method
_checkpointed_forward
Forward method with activation checkpointing.
linear_moe/model/qwen2/hybrid/hybrid_transformer_block.py:201
↓ 1 callers
Method
_checkpointed_forward
Forward method with activation checkpointing.
linear_moe/model/deepseek_v2/transformer_block.py:197
↓ 1 callers
Method
_checkpointed_forward
Forward method with activation checkpointing.
linear_moe/model/deepseek_v2/hybrid/hybrid_transformer_block.py:184
↓ 1 callers
Method
_checkpointed_forward
Forward method with activation checkpointing.
linear_moe/model/llama3/transformer_legacy.py:1611
↓ 1 callers
Method
_checkpointed_forward
Forward method with activation checkpointing.
linear_moe/model/llama3/hybrid/hybrid_transformer_block.py:201
↓ 1 callers
Method
_configure_tokenizer
Configure the text tokenizer with special tokens, whitespace tokens, and byte fallback tokens. Args: text_tokenizer (Text
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:101
↓ 1 callers
Function
_convert_ds_transformer_state
(sd_list)
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron_ori.py:49
↓ 1 callers
Function
_convert_ds_transformer_state
(sd_list)
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron.py:50
↓ 1 callers
Function
_create_checkpoint_paths
(base_folder, iteration, tp_degree, pp_degree)
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron_ori.py:59
↓ 1 callers
Function
_create_checkpoint_paths
(base_folder, iteration, tp_degree, pp_degree)
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron.py:60
↓ 1 callers
Function
_create_latest_file
(base_folder, iteration)
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron_ori.py:156
↓ 1 callers
Function
_create_latest_file
(base_folder, iteration)
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron.py:156
↓ 1 callers
Function
_create_megatron_dict
()
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron_ori.py:73
↓ 1 callers
Function
_create_megatron_dict
()
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron.py:76
↓ 1 callers
Function
_create_rank_checkpoint
(ds_checkpoint, checkpoint_path, tp_index,
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron_ori.py:98
↓ 1 callers
Function
_create_rank_checkpoint
(ds_checkpoint, tp_index, pp_index,
toolkits/model_checkpoints_convertor/bloom/deepspeed_to_megatron.py:100
↓ 1 callers
Method
_encode_whitespaces
(text: str, max_len: int = 80)
linear_moe/tokenizer/icetk_glm130b_tokenizer.py:189
↓ 1 callers
Method
_fetch_all_box_with_ref
(self, text)
linear_moe/tokenizer/tokenization_qwen_vl.py:397
↓ 1 callers
Method
_fetch_latest_picture
(self, response, history)
linear_moe/tokenizer/tokenization_qwen_vl.py:387
↓ 1 callers
Function
_get_block_submodules
( config: TransformerConfig, spec: Union[TransformerBlockSubmodules, ModuleSpec], )
linear_moe/model/mixtral/transformer_block.py:81
↓ 1 callers
Function
_get_block_submodules
( config: TransformerConfig, spec: Union[TransformerBlockSubmodules, ModuleSpec], )
linear_moe/model/qwen2/transformer_block.py:81
↓ 1 callers
Function
_get_block_submodules
( config: TransformerConfig, spec: Union[TransformerBlockSubmodules, ModuleSpec], )
linear_moe/model/deepseek_v2/transformer_block.py:82
↓ 1 callers
Method
_get_layer
(self, layer_number: int)
linear_moe/model/mixtral/transformer_block.py:206
↓ 1 callers
Method
_get_layer
(self, layer_number: int)
linear_moe/model/mixtral/hybrid/hybrid_transformer_block.py:198
↓ 1 callers
Method
_get_layer
(self, layer_number: int)
linear_moe/model/qwen2/transformer_block.py:206
↓ 1 callers
Method
_get_layer
(self, layer_number: int)
linear_moe/model/qwen2/hybrid/hybrid_transformer_block.py:198
↓ 1 callers
Method
_get_layer
(self, layer_number: int)
linear_moe/model/deepseek_v2/transformer_block.py:194
↓ 1 callers
Method
_get_layer
(self, layer_number: int)
linear_moe/model/deepseek_v2/hybrid/hybrid_transformer_block.py:181
↓ 1 callers
Method
_get_layer
(self, layer_number: int)
linear_moe/model/llama3/hybrid/hybrid_transformer_block.py:198
↓ 1 callers
Function
_get_layer_type
(model_type, default_layer_type, retro_layer_numbers, layer_number)
linear_moe/model/llama3/transformer_legacy.py:1375
↓ 1 callers
Function
_get_num_layers
Compute the number of transformer layers resident on the current rank.
linear_moe/model/llama3/transformer_legacy.py:1320
↓ 1 callers
Method
_get_states_from_cache
(self, inference_params, batch_size, initialize_states=False)
linear_moe/sequence_modeling/mamba2/mamba_mixer.py:493
↓ 1 callers
Function
_init_embedding_weights
(module)
toolkits/model_checkpoints_convertor/falcon40b/checkpoint_reshaping_and_interoperability.py:301
← previous
next →
201–300 of 1,498, ranked by callers