MCPcopy Create free account

hub / github.com/JonasGeiping/cramming / functions

Functions411 in github.com/JonasGeiping/cramming

⨍ Functions 411◇ Types & classes 86
↓ 16 callersMethod__init__
(self, hidden_size, cfg_attention, expansion_factor: int = 2, s: int = 128)
cramming/architectures/attention.py:642
↓ 16 callersFunction_get_norm_fn
(norm_name)
cramming/architectures/components.py:228
↓ 12 callersFunction_get_fake_step
(current_step, initial_time, hour_budget, num_training_steps)
cramming/backend/optimizers/schedulers.py:259
↓ 10 callersMethod__init__
(self, sub_activation)
cramming/architectures/components.py:271
↓ 9 callersFunction_maybe_record
(key, step=-1)
cramming/utils.py:232
↓ 7 callersMethod__init__
(self, param: Tensor)
cramming/backend/optimizers/shampoo/shampoo_utils.py:618
↓ 6 callersFunction_get_nonlin_fn
(nonlin_name, use_gating=True)
cramming/architectures/components.py:242
↓ 6 callersFunction_init_module
(name, module, init_method, init_std=0.02, hidden_size=768, num_layers=12)
cramming/architectures/scriptable_bert.py:428
↓ 6 callersFunctionget_extended_attention_mask
Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor
cramming/architectures/components.py:332
↓ 6 callersFunctionis_main_process
()
cramming/utils.py:163
↓ 5 callersMethod__init__
.
cramming/architectures/losses.py:84
↓ 5 callersMethod__init__
(self, dim, base=10000, def_seq_length=128, seq_dim: int = 0)
cramming/architectures/embeddings.py:77
↓ 5 callersMethodto_device
Move batch of data into device memory.
cramming/backend/torch_default.py:117
↓ 4 callersMethod__init__
(self, config)
cramming/architectures/scriptable_bert.py:51
↓ 4 callersMethodforward_step
Take another step forward on a set of given hidden states.
cramming/architectures/recurrent_transformers.py:135
↓ 4 callersMethodstep
(self, batch: dict[str, torch.Tensor])
cramming/backend/torch_default.py:111
↓ 4 callersMethodupdate_preconditioners
(self, grad: Tensor)
cramming/backend/optimizers/shampoo/shampoo_utils.py:622
↓ 3 callersMethod__init__
(self, encoder, cfg_arch)
cramming/architectures/recurrent_transformers.py:144
↓ 3 callersMethod__init__
(self, cfg_arch)
cramming/architectures/fixed_cramlm.py:60
↓ 3 callersFunction_init_module
(name, module, init_method, init_std=0.02, hidden_size=768, num_layers=12)
cramming/architectures/fixed_cramlm.py:199
↓ 3 callersMethodbroadcast
(self, src_rank: int)
cramming/backend/optimizers/shampoo/shampoo_utils.py:143
↓ 3 callersMethodforward_embed
(self, input_ids)
cramming/architectures/recurrent_transformers.py:123
↓ 3 callersFunctionget_layer_fn
(type="pre", prob=0.1, scripting=True, dn=False, drop=False)
cramming/architectures/fused_layers.py:8
↓ 3 callersFunctionload_tokenizer
Load a tokenizer from disk/huggingface. This will never construct a new tokenizer.
cramming/data/tokenizer_preparation.py:10
↓ 3 callersFunctionmain_process_first
A context manager for torch distributed environment where on needs to do something on the main process, while blocking replicas, and when it'
cramming/data/pretraining_preparation.py:421
↓ 3 callersFunctionmulti_dim_split
Chunks tensor across multiple dimensions based on splits. Args: grad (Tensor): Gradient or tensor to split. splits (List[List[int
cramming/backend/optimizers/shampoo/shampoo_utils.py:71
↓ 3 callersMethodprecondition_and_update
( self, param: Tensor, grad: Tensor, lr: Union[float, Tensor], )
cramming/backend/optimizers/shampoo/shampoo_utils.py:631
↓ 3 callersFunctionprepare_downstream_dataloader
(dataset, tokenizer, mode, cfg_impl)
cramming/backend/utils.py:116
↓ 3 callersMethodretrieve_model_state_dict
(self)
cramming/backend/torch_default.py:241
↓ 3 callersMethodrotate_half
(x: torch.Tensor)
cramming/architectures/embeddings.py:88
↓ 3 callersMethodrotate_half
(self, x: torch.Tensor)
cramming/architectures/embeddings.py:170
↓ 3 callersMethodsingle_forward
For cases where shapes of Q and K do not match.
cramming/architectures/embeddings.py:122
↓ 3 callersMethodstep
(self, closure=None)
cramming/backend/optimizers/optimizer_modifiers.py:170
↓ 2 callersMethod__init__
(self, cfg_arch, seq_length_in: int, seq_length_out: int)
cramming/architectures/funnel_transformers.py:47
↓ 2 callersMethod__iter__
(self)
cramming/backend/utils.py:262
↓ 2 callersFunction_download_tokenizer
(tokenizer_path_or_name, seq_length, cache_dir=None)
cramming/data/tokenizer_preparation.py:31
↓ 2 callersMethod_get_cos_sin
(self)
cramming/architectures/embeddings.py:109
↓ 2 callersMethod_get_cos_sin
(self)
cramming/architectures/embeddings.py:155
↓ 2 callersMethod_init_cursor
Initialize cursor position.
cramming/data/lmdb_datasets.py:108
↓ 2 callersFunction_load_optimizer
(model, cfg_train, cfg_impl)
cramming/backend/torch_default.py:439
↓ 2 callersMethod_prediction_dynamic
(self, outputs: torch.Tensor, labels: Optional[torch.Tensor] = None)
cramming/architectures/recurrent_transformers.py:255
↓ 2 callersMethod_prediction_fixed
(self, outputs: torch.Tensor, labels: Optional[torch.Tensor] = None)
cramming/architectures/recurrent_transformers.py:269
↓ 2 callersFunction_sort_tokenized_dataset_by_token
(tokenized_dataset, tokenizer, target_token_id, num_threads=1)
cramming/data/curriculum_sorting.py:57
↓ 2 callersFunction_sort_tokenized_dataset_by_unigram
(tokenized_dataset, tokenizer, num_threads=1, ngram=1, reverse=False)
cramming/data/curriculum_sorting.py:10
↓ 2 callersFunctionapply_rotary_pos_emb
(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor, seq_dimension: int = -2)
cramming/architectures/embeddings.py:244
↓ 2 callersMethodattention
tuformer attention in batch first implementation (hopefully)
cramming/architectures/attention.py:431
↓ 2 callersMethodbackward
(self, loss)
cramming/backend/torch_default.py:131
↓ 2 callersFunctionchecksum_config
This is more annoying that I thought it would be. But a json-dump of the config file is hashed and used as checksum.
cramming/data/utils.py:16
↓ 2 callersMethodcompute_root_inverse
(self)
cramming/backend/optimizers/shampoo/shampoo_utils.py:435
↓ 2 callersFunctionconstruct_tokenizer
Construct a new tokenizer. This may include downloading from huggingface.
cramming/data/tokenizer_preparation.py:21
↓ 2 callersMethodexit
(self, hidden_states)
cramming/architectures/recurrent_transformers.py:130
↓ 2 callersMethodforward
(self, *inputs, **kwargs)
cramming/backend/torch_default.py:127
↓ 2 callersMethodget_cos_sin_cache
(self, x: torch.Tensor)
cramming/architectures/embeddings.py:100
↓ 2 callersFunctionget_num_workers
(cfg_impl)
cramming/backend/utils.py:10
↓ 2 callersMethodget_rand_list
(self, seq_len)
cramming/data/generation_gibbs.py:21
↓ 2 callersFunctionget_schedule_fn
Returns a callable scheduler_fn(optimizer). Todo: Sanitize and unify these schedulers...
cramming/backend/optimizers/schedulers.py:11
↓ 2 callersFunctiongroup_parameters
(model, cfg_train)
cramming/backend/utils.py:17
↓ 2 callersFunctionlayer_structure
(states, outputs, alpha, residual_scale, prob: float = 0.1, training: bool = False)
cramming/architectures/fused_layers.py:25
↓ 2 callersMethodload_checkpoint
Load list of states from checkpoint file. Not generally compatible with any other engine?
cramming/backend/torch_default.py:279
↓ 2 callersFunctionlookup_dtype
(vocab_size)
cramming/data/cached_datasets.py:11
↓ 2 callersMethodmask_prob
(self, position, sentences)
cramming/data/generation_gibbs.py:57
↓ 2 callersMethodoptimizer_step
Requires a scheduler that is based on iterations instead of epochs.
cramming/backend/torch_default.py:145
↓ 2 callersMethodprecondition
(self, grad: Tensor)
cramming/backend/optimizers/shampoo/shampoo_utils.py:393
↓ 2 callersMethodprecondition
(self, grad: Tensor)
cramming/backend/optimizers/shampoo/shampoo_utils.py:582
↓ 2 callersMethodprecondition
(self, grad: Tensor)
cramming/backend/optimizers/shampoo/shampoo_utils.py:625
↓ 2 callersFunctionprepare_pretraining_dataloader
(dataset, tokenizer, cfg_train, cfg_impl)
cramming/backend/utils.py:73
↓ 2 callersMethodreset_sample_statistics
Allocate new tensors, old references are still required for the optimizer step.
cramming/backend/optimizers/progressive_batching.py:147
↓ 2 callersFunctionrotate_half
(x: torch.Tensor)
cramming/architectures/embeddings.py:236
↓ 2 callersFunctionsave_corpus
(path)
cramming/data/pretraining_preparation.py:69
↓ 2 callersFunctionsave_to_table
Save keys to .csv files.
cramming/utils.py:310
↓ 2 callersFunctionset_jit_instructions
Refer also https://github.com/pytorch/pytorch/blob/c90be037b46f58d2b120f46a1c466976f66817b5/torch/jit/_fuser.py#L20
cramming/utils.py:363
↓ 2 callersFunctionsimplified_layer_structure
(states, outputs, alpha, residual_scale, prob: float = 0.1, training: bool = False)
cramming/architectures/fused_layers.py:42
↓ 2 callersFunctionstage_dataset
This is a mess because our network drives are a mess. You might not need this.
cramming/data/utils.py:25
↓ 2 callersMethodstep
(self, iter_num, pos)
cramming/data/generation_gibbs.py:36
↓ 2 callersMethodsweep
(self, iter_num)
cramming/data/generation_gibbs.py:42
↓ 2 callersFunctiontorchdynamo_compile_method
(method_call, optimizer_arg=None)
cramming/backend/utils.py:61
↓ 2 callersFunctionvalidate
Evaluate on validation set.
eval.py:96
↓ 1 callersMethod__init__
Load Engine. This is the bare minimum init. The model is further traced if required.
cramming/backend/torch_default.py:43
↓ 1 callersMethod__init__
(self, *args, **kwargs)
cramming/backend/utils.py:256
↓ 1 callersMethod_broadcast_inv_preconditioners
Broadcasts inverse preconditioners.
cramming/backend/optimizers/shampoo/shampoo.py:242
↓ 1 callersFunction_build_dataloaders
Construct dataloaders according to cfg_impl settings. Validation samplers always repeat on all devices.
cramming/data/downstream_task_preparation.py:110
↓ 1 callersMethod_compute_root_inverse
Preprocesses and computes root inverse of each preconditioner. Syncs root inverse across different workers.
cramming/backend/optimizers/shampoo/shampoo.py:211
↓ 1 callersFunction_concatenate_entries
(dataset, num_entries_in_group, num_threads)
cramming/data/pretraining_preparation.py:289
↓ 1 callersFunction_construct_tokenizer
The actual generation instructions for a new tokenizer. Might make this more scriptable in the future... Follows closely along with https://huggi
cramming/data/tokenizer_preparation.py:71
↓ 1 callersMethod_drop_tokens
Drop random tokens. Hou et al., "Token Dropping for Efficient BERT Pretraining" also discuss dropping tokens based on more advanced strategies
cramming/backend/utils.py:240
↓ 1 callersFunction_finish_and_return_to_hf_dataset
For simplicity the entire new dataset has to fit into memory...
cramming/data/deduplicate.py:139
↓ 1 callersMethod_forward_dynamic
(self, outputs: torch.Tensor, labels: Optional[torch.Tensor] = None)
cramming/architectures/scriptable_bert.py:207
↓ 1 callersMethod_forward_dynamic
(self, outputs: torch.Tensor, labels: Optional[torch.Tensor] = None)
cramming/architectures/fixed_cramlm.py:130
↓ 1 callersFunction_get_layer_fn
(layer_macro_type)
cramming/architectures/components.py:219
↓ 1 callersFunction_get_loss_fn
(loss_fn_name, z_loss_factor=0.0, embedding=torch.nn.Identity())
cramming/architectures/scriptable_bert.py:376
↓ 1 callersFunction_get_sane_normalizers
original rules as in XLNET with optional modifications. force_english_keyboard is actually an ascii normalization.
cramming/data/tokenizer_preparation.py:52
↓ 1 callersFunction_get_sane_token_args
()
cramming/data/tokenizer_preparation.py:40
↓ 1 callersFunction_get_size
Compute the size of a directory path. Why is this not in the standard library?
cramming/data/utils.py:64
↓ 1 callersMethod_grad_norm
(self)
cramming/backend/optimizers/optimizer_modifiers.py:180
↓ 1 callersFunction_huggingface_preprocessing
Dataset preprocessing and tokenization. This is basically the default HF routine from https://github.com/huggingface/transformers/blob/master
cramming/data/pretraining_preparation.py:200
↓ 1 callersMethod_init_distributed
(self, model)
cramming/backend/torch_default.py:253
↓ 1 callersMethod_init_weights
(self, *args, **kwargs)
cramming/architectures/scriptable_bert.py:84
↓ 1 callersMethod_init_weights
(self, *args, **kwargs)
cramming/architectures/scriptable_bert.py:171
↓ 1 callersMethod_init_weights
(self, module=None)
cramming/architectures/scriptable_bert.py:240
↓ 1 callersMethod_init_weights
(self, module=None)
cramming/architectures/scriptable_bert.py:317
next →1–100 of 411, ranked by callers