Functions — github.com/AnswerDotAI/ModernBERT

CodeHub MCPcopy Create free account

hub / github.com/AnswerDotAI/ModernBERT / functions

Functions560 in github.com/AnswerDotAI/ModernBERT

⨍ Functions 560◇ Types & classes 136↳ Endpoints 2

↓ 39 callersFunctionbuild_dataloader

(dataset, collate_fn=None, **kwargs)

src/evals/finetuning_jobs.py:55

↓ 29 callersFunctioninit_weights

Initialize weights of a linear or embedding module. :param config: The model config. :param module: The linear or embedding submodule to

src/bert_layers/initialization.py:68

↓ 27 callersMethodupdate

Updates the internal state with results from a new batch. Args: loss (~torch.Tensor): A Tensor of loss values to compare against.

src/flex_bert.py:125

↓ 22 callersFunctionget_norm_layer

(config: FlexBertConfig, compiled_norm: bool = False)

src/bert_layers/normalization.py:99

↓ 17 callersFunctioncreate_glue_dataset

src/evals/data.py:128

↓ 15 callersFunctiontile_linear

Tile the weights of a linear layer to a new, larger linear dimension. Args: pretrained_linear (nn.Linear): The original linear layer

src/bert_layers/initialization.py:388

↓ 14 callersFunctiontile_weight

Tile or center an input tensor to a larger desired size. Works for both 2D and 1D tensors. Args: pretrained_weights (torch.Tensor): The

src/bert_layers/initialization.py:172

↓ 13 callersMethod__init__

src/bert_layers/layers.py:227

↓ 12 callersMethod__init__

( self, config, add_pooling_layer: bool = True, )

src/bert_layers/model.py:173

↓ 11 callersMethod__init__

src/bert_layers/attention.py:191

↓ 11 callersMethodreset_parameters

src/bert_layers/layers.py:275

↓ 11 callersFunctiontile_list_to_length

benchmark.py:253

↓ 11 callersFunctiontry_batch_size

efficiency/multiprocess_bench.py:125

↓ 11 callersFunctionunpad_input

Remove padding from input sequences. Arguments: hidden_states: (batch, seqlen, ...) attention_mask: (batch, seqlen), bool / int,

src/bert_padding.py:79

↓ 10 callersFunctioncreate_superglue_dataset

src/evals/data.py:134

↓ 9 callersFunction_count_parameters

(model: nn.Module, trainable: bool = True)

src/bert_layers/model.py:123

↓ 9 callersMethod_init_weights

(self, reset_params: bool = False)

src/bert_layers/layers.py:269

↓ 8 callersMethod_init_weights

(self, module: Optional[nn.Module] = None, reset_params: Optional[bool] = None)

src/bert_layers/model.py:971

↓ 8 callersFunctionsafe_get

(dict_obj, key, default=None)

generate_eval_config.py:121

↓ 7 callersMethod__init__

( self, model: ComposerModel, tokenizer_name: str, job_name: Optional[str] = N

src/evals/glue_jobs.py:110

↓ 7 callersMethodbackward

(ctx, grad_output: torch.Tensor)

src/bert_padding.py:42

↓ 7 callersFunctionget_act_fn

(config: Union[FlexBertConfig, str])

src/bert_layers/activation.py:51

↓ 7 callersFunctionget_attention_layer

(config: FlexBertConfig, layer_id: Optional[int] = None)

src/bert_layers/attention.py:1545

↓ 7 callersFunctionget_mlp_layer

(config: FlexBertConfig, layer_id: Optional[int] = None)

src/bert_layers/mlp.py:200

↓ 6 callersMethod__init__

( self, model: ComposerModel, tokenizer_name: str, job_name: Optional[str] = N

src/evals/superglue_jobs.py:125

↓ 6 callersMethod__init__

( self, model: ComposerModel, tokenizer_name: str, job_name: Optional[str] = N

src/evals/misc_jobs.py:43

↓ 6 callersFunctioncompare_structures

(s1, s2, rtol=1e-5, atol=1e-8)

tests/test_sequence_packer.py:64

↓ 6 callersFunctioncreate_eval_dataset

( task: str, tokenizer_name: str, split: str, dataset_name: str, max_seq_length: int = 256

src/evals/data.py:56

↓ 6 callersFunctioncreate_mlmmlu_dataset

src/evals/data.py:168

↓ 5 callersMethodfrom_composer

Load from pre-trained.

src/bert_layers/model.py:317

↓ 5 callersFunctiongc_cuda

Garbage collect Torch (CUDA) memory.

src/callbacks/scheduled_gc.py:15

↓ 5 callersFunctionmain

(cfg: DictConfig, return_trainer: bool = False, do_train: bool = True)

↓ 5 callersFunctionpad_input

Add padding to sequences. Arguments: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask.

src/bert_padding.py:128

↓ 5 callersFunctiontile_norm

Tile the weights of a pretrained norm layer to a new, larger layer norm dimension. Args: pretrained_norm (Union[nn.LayerNorm, RMSNor

src/bert_layers/initialization.py:485

↓ 4 callersMethod__init__

(self, config: FlexBertConfig, layer_id: Optional[int] = None)

src/bert_layers/mlp.py:98

↓ 4 callersMethod__init__

src/bert_layers/embeddings.py:40

↓ 4 callersMethod_init_module_weights

Custom weight init of modules using src.bert_layers.initialization.init_weights Currently only supports init of embedding modules

src/bert_layers/model.py:868

↓ 4 callersFunctionbatched

tests/test_sequence_packer.py:25

↓ 4 callersFunctioncheck_finished_jobs

Check for finished jobs and free up their GPUs.

run_evals.py:225

↓ 4 callersMethodget_number_parameters

Returns the number of parameters in the model. Args: count_embeddings: count the parameters in the embeddings layer, excluding po

src/bert_layers/model.py:986

↓ 4 callersMethodpost_init

src/bert_layers/model.py:934

↓ 4 callersMethodrun

Trains the model, optionally pulling a GPU id from the queue. Returns: A dict with keys: * 'checkpoints': list of sav

src/evals/finetuning_jobs.py:149

↓ 4 callersFunctiontokenize_and_write

(writer, pool, tokenizer)

src/data/sample_dataset_from_config.py:21

↓ 3 callersFunction_print_averaged_glue_results

Pretty prints a table of glue results averaged across seeds.

↓ 3 callersFunction_print_averaged_glue_results

Pretty prints a table of glue results averaged across seeds.

↓ 3 callersFunction_raise_if_schedule_and_max_incompatible

Checks that t_schedule and t_max have the same units. _convert_time should be called on both `t_warmup` and `t_max` before this function is calle

src/scheduler.py:22

↓ 3 callersMethodcompute

src/evals/superglue_jobs.py:115

↓ 3 callersFunctionget_values_from_path

Parses out information from a path/string that looks like. ...<separator>key=value<separator...

↓ 3 callersFunctionget_values_from_path

Parses out information from a path/string that looks like. ...<separator>key=value<separator...

↓ 3 callersFunctionmaybe_add_padding

(config: FlexBertConfig, config_option: str)

src/bert_layers/configuration_bert.py:262

↓ 3 callersMethodreset_parameters

src/bert_layers/embeddings.py:116

↓ 3 callersMethodreset_parameters

src/bert_layers/model.py:983

↓ 3 callersFunctionrun_job

Run a job with optional GPU management.

run_evals.py:182

↓ 3 callersFunctiontile_fused_glu

Tile the weights of a fused pretrained GLU layer to a new, larger GLU dimension. Args: pretrained_glu_weight (torch.Tensor): The ori

src/bert_layers/initialization.py:314

↓ 3 callersFunctiontile_fused_qkv

Tile the weights of a fused pretrained QKV layer to a new, larger QKV dimension. Args: pretrained_qkv_weight (torch.Tensor): The ori

src/bert_layers/initialization.py:286

↓ 3 callersFunctiontrain

Main training logic. Args: config (DictConfig): Configuration composed by OmegaConf

↓ 3 callersFunctiontrain

Main training logic. Args: config (DictConfig): Configuration composed by OmegaConf

↓ 2 callersMethod__init__

(self, dist_sync_on_step: bool = False)

src/flex_bert.py:120

↓ 2 callersMethod__init__

(self, dataset_name: str, data_subset: Union[str, None], split: str)

src/convert_dataset.py:149

↓ 2 callersMethod_calculate_increase_every_tokens

src/algorithms/rope_schedule.py:60

↓ 2 callersMethod_compute_inv_freq

(self, device=None)

src/bert_layers/rotary.py:216

↓ 2 callersFunction_get_scheduler

(scheduler_type: Schedule)

src/scheduler.py:76

↓ 2 callersFunction_inverse_sqrt_schedule

Implements an inverse square root curve. Curve is alpha / sqrt(x + beta). Additionally, param x is clipped to the interval [0, inf)

src/scheduler.py:68

↓ 2 callersMethod_update_cos_sin_cache

(self, seqlen, device=None, dtype=None)

src/bert_layers/rotary.py:219

↓ 2 callersMethod_warmup_cooldown_warning

(self, type: str)

src/scheduler.py:254

↓ 2 callersFunctionapply_rotary_emb_unpad

Arguments: qkv: (total_nnz, 3, nheads, headdim) - input tensor for packed QKV. cos, sin: (seqlen_rotary, rotary_dim / 2)

src/bert_layers/rotary.py:130

↓ 2 callersFunctionbuild_dataloader

( cfg, tokenizer, device_batch_size, count_padding_tokens=True, device_microbatch_size: in

↓ 2 callersFunctionbuild_model

(cfg: DictConfig)

↓ 2 callersFunctionbuild_my_dataloader

Create a dataloader for classification. **Modify this function to train on your own dataset!** This function is provided as a starter code t

sequence_classification.py:149

↓ 2 callersFunctioncreate_eurlex_dataset

src/evals/data.py:152

↓ 2 callersFunctioncreate_job_configs

( main_config: om.DictConfig, tasks_to_run: Set[str], pretrained_checkpoint_path: Optional[str], )

↓ 2 callersFunctioncreate_job_configs

( main_config: om.DictConfig, tasks_to_run: Set[str], pretrained_checkpoint_path: Optional[str], )

↓ 2 callersFunctioncreate_swag_dataset

src/evals/data.py:142

↓ 2 callersFunctioncreate_ultrafeedback_dataset

src/evals/data.py:160

↓ 2 callersFunctionget_bert_layer

(config: FlexBertConfig, layer_id: Optional[int] = None)

src/bert_layers/layers.py:572

↓ 2 callersFunctionget_free_gpu

Check for free GPUs, prioritizing potentially free GPUs.

run_evals.py:113

↓ 2 callersFunctionget_gpu_power

benchmark.py:128

↓ 2 callersFunctionget_model

( hidden_size: int, num_hidden_layers: int, intermediate_size: float, parallel_attn: bool = Tr

benchmark.py:45

↓ 2 callersFunctionhandle_process_completion

Handles the completion of a process, checks for errors, cleans up stderr_file, and logs messages.

run_evals.py:146

↓ 2 callersMethodmlm_masking

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. This is exactly a numpy versio

src/sequence_packer.py:284

↓ 2 callersFunctionmove_and_flatten_files

(local_dir: Path)

run_evals.py:552

↓ 2 callersMethodpad_inputs

( self, inputs: torch.Tensor, indices: torch.Tensor, batch_size: int,

src/bert_layers/model.py:1079

↓ 2 callersFunctionparam_groups_weight_decay

(model: nn.Module, weight_decay=1e-5, no_weight_decay_list=())

↓ 2 callersFunctionpush_to_hub_incrementally

(repo_name, local_path)

src/data/hf_to_mds.py:31

↓ 2 callersMethodrebuild_alibi_tensor

(self, size: int, device: Optional[Union[torch.device, str]] = None)

src/bert_layers/layers.py:98

↓ 2 callersFunctionrun_jobs_parallel

Runs a list of jobs (passed in as Hydra configs) across GPUs. Returns a dictionary mapping job name to the result and original config Each jo

↓ 2 callersFunctionrun_jobs_parallel

Runs a list of jobs (passed in as Hydra configs) across GPUs. Returns a dictionary mapping job name to the result and original config Each jo

↓ 2 callersFunctionrun_jobs_serial

Runs the jobs serially, rather than in parallel. Useful for debugging

↓ 2 callersFunctionrun_jobs_serial

Runs the jobs serially, rather than in parallel. Useful for debugging

↓ 2 callersFunctionsplit_packed_batch

(batch: Any, microbatch_size: Union[int, float], padding_tolerance=1.0)

src/sequence_packer.py:534

↓ 2 callersMethodstep

Performs a single optimization step on the whole model or individual parameter. Args: closure: A closure which reevaluates the mo

src/optimizer.py:139

↓ 2 callersFunctiontile_embedding

Tile the weights of an embedding layer to a new, larger embedding dimension. Args: pretrained_embedding (nn.Embedding): The original emb

src/bert_layers/initialization.py:515

↓ 2 callersFunctiontile_fused_qkvff

Tile the weights of a fused pretrained QKVFF layer to a new, larger QKVFF dimension. Args: pretrained_qkvff_weight (torch.Tensor): T

src/bert_layers/initialization.py:341

↓ 2 callersFunctionupdate_progress_for_finished_jobs

Update progress bars for any finished GPU jobs.

run_evals.py:265

↓ 1 callersMethod__getitem__

(self, index: int)

src/text_data.py:496

↓ 1 callersMethod__init__

( self, local: str, split: Optional[str], max_seq_len: int, tokenizer:

src/text_data.py:454

↓ 1 callersMethod__init__

Configuration class for MosaicBert. Args: alibi_starting_size (int): Use `alibi_starting_size` to determine how large of an alibi

src/bert_layers/configuration_bert.py:10

↓ 1 callersMethod__init__

( self, job_name: Optional[str] = None, load_path: Optional[str] = None, save_

src/evals/finetuning_jobs.py:112

↓ 1 callersMethod_calculate_step_thresholds

src/sequence_packer.py:40

↓ 1 callersFunction_cosine_schedule

Implements a cosine curve. Curve is cos(x) on domain [0, pi], stretched to the domain [0, 1] and range [start_y, finish_y]. Additionally, param x

src/scheduler.py:58

next →1–100 of 560, ranked by callers