Functions — github.com/OpenSparseLLMs/Linear-MoE

hub / github.com/OpenSparseLLMs/Linear-MoE / functions

Functions1,498 in github.com/OpenSparseLLMs/Linear-MoE

⨍ Functions 1,498◇ Types & classes 227

↓ 138 callersMethodsplit

(self, json_line)

toolkits/pretrain_data_preprocessing/preprocess_data_megatron.py:55

↓ 69 callersFunctionget_args

()

toolkits/pretrain_data_preprocessing/preprocess_data.py:72

↓ 56 callersMethodsave

(self, filepath)

linear_moe/tokenizer/tokenization_qwen_vl.py:475

↓ 33 callersFunction_get_mlp_module_spec

( use_te: bool = True, num_experts: int = None, moe_grouped_gemm: bool = False )

linear_moe/model/deepseek_v2/layer_specs.py:1150

↓ 26 callersFunction_get_mlp_module_spec

Helper function to create a module specification for an MLP or MoE layer. Args: use_te: Optional; if `True`, uses Transformer Engine

linear_moe/model/mixtral/layer_specs.py:1515

↓ 22 callersFunction_broadcast

(item)

linear_moe/data/utils.py:32

↓ 21 callersFunctionapply_rotary_pos_emb

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

linear_moe/model/deepseek_v2/yarn_rotary_pos_embedding.py:174

↓ 21 callersFunctionget_tokenizer

Return tokenizer.

linear_moe/tokenizer/__init__.py:34

↓ 20 callersMethoddecode

(self, token_ids)

linear_moe/tokenizer/__init__.py:211

↓ 19 callersMethodencode

(self, text)

linear_moe/tokenizer/__init__.py:208

↓ 17 callersFunction_get_mlp_module_spec

( use_te: bool = True, num_experts: int = None, moe_grouped_gemm: bool = False )

linear_moe/model/qwen2/layer_specs.py:1076

↓ 17 callersFunctionbuild_tokenizer

(args)

linear_moe/tokenizer/__init__.py:38

↓ 17 callersFunctionget_args

()

toolkits/pretrain_data_preprocessing/preprocess_data_megatron.py:175

↓ 16 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/llama/hf2megatron.py:343

↓ 15 callersFunction_get_mlp_module_spec

( use_te: bool = True, num_experts: int = None, moe_grouped_gemm: bool = False )

linear_moe/model/llama3/layer_specs.py:945

↓ 15 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/mistral/hf2megatron.py:333

↓ 15 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/baichuan/checkpoint_reshaping_and_interoperability.py:327

↓ 15 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.5.py:362

↓ 15 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/baichuan2/checkpoint_reshaping_and_interoperability.py:314

↓ 14 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.0.py:339

↓ 12 callersMethod__init__

( self, head_dim: int, dot_dim: int = None, bias: Optional[bool] = False )

linear_moe/model/common_modules/feature_map.py:73

↓ 12 callersFunctionget_element_from_dict_by_path

(d, path)

toolkits/model_checkpoints_convertor/llama/hf2mcore_70b.py:264

↓ 12 callersFunctionget_element_from_dict_by_path

(d, path)

toolkits/model_checkpoints_convertor/mistral/hf2mcore_mixtral.py:346

↓ 12 callersFunctionget_element_from_dict_by_path

(d, path)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_gqa.py:243

↓ 12 callersFunctioninitialize_megatron

Set global variables, initialize distributed, and set autoresume and random seeds. `allow_no_cuda` should not be set unless using megatron for

linear_moe/initialize.py:20

↓ 11 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/falcon40b/checkpoint_reshaping_and_interoperability.py:287

↓ 11 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/falcon/checkpoint_reshaping_and_interoperability.py:289

↓ 11 callersFunctionlog_single_rank

(logger: logging.Logger, *args: Any, rank: int = 0, **kwargs: Any)

linear_moe/sequence_modeling/mamba2/mamba_hybrid_layer_allocation.py:10

↓ 10 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/bloom/checkpoint_reshaping_and_interoperability.py:287

↓ 10 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/bloom/reward_model_to_megatron.py:287

↓ 9 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/starcoder/checkpoint_reshaping_and_interoperability.py:309

↓ 8 callersMethod__init__

(self, drop_prob=0.)

linear_moe/model/llama3/transformer_legacy.py:76

↓ 8 callersMethodadd_special_tokens

Add a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current vocabula

linear_moe/tokenizer/icetk_glm130b_tokenizer.py:286

↓ 8 callersFunctioncore_transformer_config_from_args

(args, config_class=None)

linear_moe/arguments.py:23

↓ 8 callersFunctionget_patch_args

(parser)

linear_moe/arguments.py:64

↓ 8 callersMethodstate_dict_for_save_checkpoint

(self, prefix='', keep_vars=False)

linear_moe/model/llama3/gpt_model.py:112

↓ 7 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/baichuan/hf2te.py:194

↓ 7 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/yi/checkpoint_reshaping_and_interoperability.py:314

↓ 7 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/baichuan2/hf2te.py:178

↓ 7 callersMethodload_state_dict

Customized load.

linear_moe/model/llama3/gpt_model.py:125

↓ 7 callersFunctionmodel_provider

Builds the model. If you set the use_mcore_models to True, it will return the mcore GPT model and if not the legacy GPT model. Args:

examples/linear_llama3/pretrain_llama.py:55

↓ 7 callersMethodtokenize

Tokenize a list of strings. Args: strings (List[str]): a list of strings to be tokenized tokenizer (Tokenizer

linear_moe/data/llama.py:178

↓ 6 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/glm/checkpoint_reshaping_and_interoperability.py:257

↓ 6 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/galactica/checkpoint_reshaping_and_interoperability.py:316

↓ 6 callersFunctionsave_state_dict

(args, model, checkpoint_name)

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py:444

↓ 5 callersMethod__init__

( self, hidden_size: int, elementwise_affine: bool = True, bias: bool = False,

linear_moe/model/common_modules/layernorm.py:628

↓ 5 callersMethod_get_text_tokenizer

(self, encode_special_tokens=False)

linear_moe/tokenizer/icetk_glm130b_tokenizer.py:143

↓ 5 callersFunction_vocab_size_with_padding

Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.

linear_moe/tokenizer/__init__.py:17

↓ 5 callersMethodcopy

(self)

linear_moe/data/llava/conversation.py:204

↓ 5 callersFunctionget_batch_on_this_tp_rank_original

(data_iterator)

linear_moe/data/utils.py:29

↓ 5 callersFunctionget_gpt_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:80

↓ 5 callersFunctionget_gpt_layer_with_transformer_engine_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:51

↓ 5 callersFunctionget_model

Build the model.

linear_moe/training.py:184

↓ 5 callersMethodload_state_dict

Customized load.

linear_moe/model/llama3/language_model.py:267

↓ 5 callersFunctionmegatron_to_transformers_fix_query_key_value_ordering

Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :] for compatibility with later versions of NVIDIA Megatron-LM. The

toolkits/model_checkpoints_convertor/llama/hf2megatron.py:227

↓ 5 callersMethodnormalize

Normalize an image. image = (image - image_mean) / image_std. Args: image (`np.ndarray`): Image to norma

linear_moe/data/llava/cvcuda_image_processing_clip.py:206

↓ 5 callersMethodpad

(self)

linear_moe/tokenizer/icetk_glm130b_tokenizer.py:69

↓ 4 callersFunctionbuild_data_loader

Data loader. Note that batch-size is the local (per GPU) batch-size.

linear_moe/finetune_utils.py:82

↓ 4 callersFunctionbuild_pretrain_dataset_from_original

(dataset)

linear_moe/data/__init__.py:79

↓ 4 callersMethodelu

(x)

linear_moe/sequence_modeling/retention/retention.py:34

↓ 4 callersFunctionevaluate_and_print_results

Helper function to evaluate and dump results on screen.

linear_moe/training.py:782

↓ 4 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/glm130b/checkpoint_reshaping_and_interoperability.py:257

↓ 4 callersFunctionget_element_from_dict_by_path

Get element from dictionary by path. If element is not present, recursively add empty dictionaries. Args: d (dict): the dictionary to

toolkits/model_checkpoints_convertor/chatglm/checkpoint_reshaping_and_interoperability.py:284

↓ 4 callersFunctionget_hybrid_based_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:246

↓ 4 callersFunctionget_hybrid_basic_linear_attention_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:816

↓ 4 callersFunctionget_hybrid_deltanet_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:501

↓ 4 callersFunctionget_hybrid_gated_deltanet_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:564

↓ 4 callersFunctionget_hybrid_gla_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:372

↓ 4 callersFunctionget_hybrid_hgrn2_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:1012

↓ 4 callersFunctionget_hybrid_mamba2_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:113

↓ 4 callersFunctionget_hybrid_mom_gated_deltanet_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:627

↓ 4 callersFunctionget_hybrid_mom_gla_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:437

↓ 4 callersFunctionget_hybrid_rebased_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:309

↓ 4 callersFunctionget_hybrid_retention_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:183

↓ 4 callersFunctionget_hybrid_rwkv6_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:879

↓ 4 callersFunctionget_hybrid_rwkv7_linear_moe_layer_local_spec

( num_experts: int = None, moe_grouped_gemm: bool = False, qk_layernorm: bool = False )

linear_moe/model/qwen2/layer_specs.py:944

↓ 4 callersFunctionmegatron_to_transformers_fix_query_key_value_ordering

Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :] for compatibility with later versions of NVIDIA Megatron-LM. The

toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen1.5_dense_gqa.py:150

↓ 4 callersFunctionmegatron_to_transformers_fix_query_key_value_ordering

Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :] for compatibility with later versions of NVIDIA Megatron-LM. The

toolkits/model_checkpoints_convertor/qwen/hf2megatron_qwen1.5.py:246

↓ 4 callersFunctionpretrain

Main training program. Refer to https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/training.py This function will run the followings in

linear_moe/training.py:51

↓ 4 callersMethodresize

Resize an image. The shortest edge of the image is resized to size["shortest_edge"], with the longest edge resized to keep the input

linear_moe/data/llava/cvcuda_image_processing_clip.py:134

↓ 4 callersFunctionsave_checkpoint_and_time

(iteration, model, optimizer, opt_param_scheduler)

linear_moe/training.py:557

↓ 4 callersFunctionsave_state_dict

(args, model, checkpoint_name)

toolkits/model_checkpoints_convertor/llama/hf2mcore.py:434

↓ 4 callersFunctionsave_state_dict

(args, model, checkpoint_name)

toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:359

↓ 4 callersFunctionsave_state_dict

(args, model, checkpoint_name)

toolkits/model_checkpoints_convertor/qwen/hf2megablocks_qwen1.5.py:454

↓ 4 callersFunctionsinkhorn

Sinkhorn based MoE routing function Args: cost: A 2D tensor representing the cost matrix to be normalized. tol: A

linear_moe/model/mixtral/moe/moe_utils.py:51

↓ 4 callersMethodstate_dict_for_save_checkpoint

For easy load.

linear_moe/model/llama3/language_model.py:249

↓ 4 callersFunctiontokenizer_image_token

(prompt, tokenizer, image_token_index=IMAGE_TOKEN_INDEX, return_tensors=None)

linear_moe/data/llava/mm_utils.py:56

↓ 4 callersFunctionz_loss_func

Encourages the router's logits to remain small to enhance stability. Please refer to the ST-MoE paper (https://arxiv.org/pdf/2202.08906.pdf) for d

linear_moe/model/mixtral/moe/moe_utils.py:36

↓ 3 callersMethod__init__

(self, tokenizer_path, extra_vocab_size)

linear_moe/tokenizer/__init__.py:173

↓ 3 callersFunction_build_attention_mask_and_position_ids

Build the attention mask and postition ids for the input tokens.

linear_moe/generation/generation.py:518

↓ 3 callersMethod_get_layer_offset

(self)

linear_moe/model/mixtral/transformer_layer.py:153

↓ 3 callersMethod_get_layer_offset

(self)

linear_moe/model/qwen2/transformer_layer.py:153

↓ 3 callersMethod_get_layer_offset

(self)

linear_moe/model/deepseek_v2/transformer_layer.py:154

↓ 3 callersFunction_replace_closed_tag

( input_tokens: List[Any], start_tags: Union[Any, Tuple[Any]], end_tags: Union[Any, Tuple[Any]],

linear_moe/tokenizer/tokenization_qwen_vl.py:63

↓ 3 callersFunctionallocate_layers

( total_layers_count: int, target_attention_ratio: float, target_mlp_ratio: float, override_pa

linear_moe/sequence_modeling/mamba2/mamba_hybrid_layer_allocation.py:84

↓ 3 callersFunctionbias_dropout_add

(x, bias, residual, prob, training)

linear_moe/model/llama3/transformer_legacy.py:832

↓ 3 callersFunctionbuild_dataset

(index, name)

linear_moe/data/__init__.py:206

↓ 3 callersFunctioncheck_files_exist

(in_ss_out_names, key, num_partitions)

toolkits/pretrain_data_preprocessing/preprocess_data_megatron.py:271

↓ 3 callersFunctioncopy_huggingface_tokenizer

(src_path, dst_path, with_code=False)

toolkits/model_checkpoints_convertor/llama/hf2mcore.py:203

↓ 3 callersFunctioncopy_huggingface_tokenizer

(src_path, dst_path, with_code=False)

toolkits/model_checkpoints_convertor/mistral/hf2mcore.py:162

next →1–100 of 1,498, ranked by callers