MCPcopy Create free account

hub / github.com/KellerJordan/modded-nanogpt / functions

Functions2,320 in github.com/KellerJordan/modded-nanogpt

↓ 2 callersMethodupdate_preconditioner
Updates the preconditioner matrices and the eigenbases (L, R, Q_L, Q_R in the paper).
records/track_1_short/2024-10-09_SOAP/train_gpt2.py:264
↓ 2 callersMethodwrap_model
( self, model: nn.Module, local_rank: int, backend: str, device_mesh:
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/examples/parallelism.py:106
↓ 2 callersMethodwrap_model
( self, model: nn.Module, local_rank: int, backend: str, device_mesh:
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/examples/parallelism.py:136
↓ 2 callersFunctionwrite_datafile
Saves token data as a .bin file, for reading in C. - First comes a header with 256 int32s - The tokens follow, each as a uint16
data/fineweb.py:28
↓ 2 callersFunctionzeropower_frobenius_norm_like
(G: Tensor)
records/track_3_optimization/results/20260520_rre_extrapolation_pr300_2925/train_gpt_simple_rre_pr300_2925.py:351
↓ 2 callersFunctionzeropower_frobenius_norm_like
(G: Tensor)
records/track_3_optimization/results/20260529_tail_phase_readout_2850/train_gpt_tail_phase_readout_2850.py:406
↓ 2 callersFunctionzeropower_frobenius_norm_like
(G: Tensor)
records/track_3_optimization/results/20260520_tail_refinterp_2900/train_gpt_tail_refinterp_2900.py:493
↓ 2 callersFunctionzeropower_via_newtonschulz5
(G: Tensor)
records/track_3_optimization/results/20260519_dynmuon/train_gpt_simple_dynmuon.py:192
↓ 1 callersFunctionXTX
Launch Triton kernel to compute C = A.T @ A For tall matrices (M > K), this is more efficient than transposing and using XXT because
triton_kernels.py:227
↓ 1 callersFunctionXXT
Launch Triton kernel to compute C = A @ A.T
train_gpt_medium.py:229
↓ 1 callersFunctionXXT
Launch Triton kernel to compute C = A @ A.T
triton_kernels.py:100
↓ 1 callersFunctionXXT
Launch Triton kernel to compute C = A @ A.T
records/track_1_short/2025-12-11_NorMuonOptimsAndFixes/profiler-example-traces/train_gpt-profiler-example.py:233
↓ 1 callersMethod__enter__
(self)
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/utils/shampoo_utils.py:260
↓ 1 callersMethod__exit__
( self, exc_type: type[BaseException] | None, exc_val: BaseException | None, e
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/utils/shampoo_utils.py:264
↓ 1 callersMethod__init__
( self, quantized_values: Tensor, block_info: BlockInfo, min_value: Tensor | N
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/utils/shampoo_quantization.py:36
↓ 1 callersMethod__init__
( self, block_list: tuple[Tensor, ...], state: Mapping[Tensor, _StateValueType],
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/shampoo_preconditioner_list.py:1077
↓ 1 callersMethod_adam_update
Apply Adam update to a parameter. Returns the updated p_slice.
train_gpt.py:821
↓ 1 callersMethod_adam_update_step
Compiled Adam update step.
train_gpt.py:849
↓ 1 callersMethod_add_l2_regularization
( self, state_lists: dict[str, Any], weight_decay: float, weight_decay_type: W
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:1018
↓ 1 callersMethod_amortized_computation
Performs computationally expensive matrix operations for Shampoo preconditioners. This method handles the heavy computational work that is to
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/shampoo_preconditioner_list.py:176
↓ 1 callersMethod_apply_decoupled_or_corrected_weight_decay
( self, state_lists: dict[str, Any], masked_blocked_search_directions: tuple[torch.Ten
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:1093
↓ 1 callersMethod_apply_in_place_primal_averaging
( self, state_lists: dict[str, Any], masked_blocked_search_directions: tuple[torch.Ten
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:1167
↓ 1 callersMethod_apply_normuon_variance_reduction
NorMuon variance reduction. Algebraically fuses the normalization steps to minimize memory ops.
train_gpt.py:929
↓ 1 callersMethod_build_mtp_schedule
(self)
train_gpt_medium.py:1505
↓ 1 callersMethod_build_param_cfg
Build config for a single parameter from param_table.
train_gpt.py:467
↓ 1 callersMethod_circuit_muon_step
Circuit-Muon coupled update for each (attn.v, attn.proj) pair. Computed redundantly on every rank — params are replicated and grads are
records/track_3_optimization/results/20260529_tail_phase_readout_2850/train_gpt_tail_phase_readout_2850.py:831
↓ 1 callersMethod_compute_filtered_grad_list
( self, state_lists: dict[str, Any], step: torch.Tensor, beta1: float,
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:1053
↓ 1 callersMethod_compute_outer_product_list
( self, grad: Tensor, order: int, preconditioned_dims_selector: tuple[bool, ..
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/shampoo_preconditioner_list.py:1321
↓ 1 callersMethod_compute_preconditioned_gradient
Applies the Shampoo preconditioner to a gradient tensor. This method is implemented by subclasses to perform the actual precondition
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/shampoo_preconditioner_list.py:1472
↓ 1 callersMethod_compute_search_directions
( self, state_lists: dict[str, Any], step: torch.Tensor, lr: torch.Tensor,
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:1212
↓ 1 callersMethod_construct_distributed_buffers
Construct the distributed buffers for AllGather communications. Note that this function will construct the distributed buffer for the AllGath
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_ddp_distributor.py:456
↓ 1 callersMethod_construct_distributed_buffers
Construct the distributed buffers for AllGather communications. Note that this function will construct the distributed buffer for the AllGath
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_hsdp_distributor.py:459
↓ 1 callersMethod_construct_distributed_buffers
Construct the distributed buffers for AllGather communications. Note that this function will construct the distributed buffer for the AllGath
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_hybrid_shard_distributor.py:472
↓ 1 callersMethod_construct_local_block_info_list
Construct the local block info list. This method creates a list of DTensorBlockInfo objects, which contain information about each parameter b
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_ddp_distributor.py:362
↓ 1 callersMethod_construct_local_block_info_list
Construct the local block info list. This method creates a list of DTensorBlockInfo objects, which contain information about each parameter b
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_hsdp_distributor.py:316
↓ 1 callersMethod_construct_local_block_info_list
Construct the local block info list. This method creates a list of BlockInfo objects, which contain information about each parameter block,
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_distributor.py:391
↓ 1 callersMethod_construct_local_block_info_list
Construct the local block info list.
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_hybrid_shard_distributor.py:403
↓ 1 callersMethod_construct_local_block_info_list_with_params
Construct the local block info list. This method creates a list of DTensorBlockInfo objects, which contain information about each parameter b
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_hybrid_shard_distributor.py:353
↓ 1 callersMethod_construct_model
( post_model_decoration: Callable[[nn.Module], nn.Module] = lambda x: x, distributed_config: F
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/gpu_tests/shampoo_fsdp_distributor_test.py:65
↓ 1 callersMethod_create_kronecker_factors_state
( self, block_list: tuple[Tensor, ...], state: Mapping[Tensor, _StateValueType],
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/shampoo_preconditioner_list.py:1143
↓ 1 callersMethod_create_preconditioned_dims_selector
Creates a preconditioned dimensions selectors for a block. Args: dims (torch.Size): The dimensions of the block.
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/shampoo_preconditioner_list.py:1130
↓ 1 callersFunction_eta
(progress: float, cooldown_frac: float, power: float = 1.0, floor: float = 0.0)
records/track_3_optimization/results/20260518_soaph/train_gpt_simple_soaph.py:520
↓ 1 callersMethod_expected_local_block_info_list
Returns expected block info list used in test_local_block_info_list
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/gpu_tests/distributor_test_utils.py:126
↓ 1 callersMethod_flatten_params
(self, params)
records/track_3_optimization/results/20260520_rre_extrapolation_pr300_2925/train_gpt_simple_rre_pr300_2925.py:1059
↓ 1 callersMethod_get_composable_block_id_rank
Return the rank to use for composable block IDs. For shard dependent hybrid shard distributor, we need to include the rank within each sharde
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_hybrid_shard_distributor.py:340
↓ 1 callersFunction_get_dummy_f32
(device)
triton_kernels.py:477
↓ 1 callersMethod_get_inverse_roots_from_override
Retrieves the inverse roots from the override parameter for a block. For a block, we compute the inverse root from the inverse expon
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/shampoo_preconditioner_list.py:1211
↓ 1 callersMethod_get_params_or_grads
( self, get_grad: Literal[True] )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_ddp_distributor.py:224
↓ 1 callersMethod_get_params_or_grads
( self, get_grad: Literal[True] )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_fully_shard_distributor.py:33
↓ 1 callersMethod_get_params_or_grads
( self, get_grad: Literal[True] )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/_shampoo_hybrid_shard_lossless_distributor.py:127
↓ 1 callersMethod_get_params_or_grads
( self, get_grad: Literal[True] )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/_shampoo_fully_shard_lossless_distributor.py:121
↓ 1 callersMethod_get_params_or_grads
( self, get_grad: Literal[True] )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_hybrid_shard_distributor.py:222
↓ 1 callersMethod_infer_conv_output_shape
( input_shape: tuple[int, ...], kernel_size: int, stride: int, padding: int )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/examples/convnet.py:64
↓ 1 callersMethod_init_state
Initialize optimizer state for all parameters.
train_gpt.py:538
↓ 1 callersMethod_init_state
(self, p)
records/track_3_optimization/results/20260525_aurora_ema_ref/train_gpt_simple_aurora_ema_ref.py:427
↓ 1 callersMethod_init_state
(self, p: Tensor, state: dict)
records/track_3_optimization/results/20260518_soaph/train_gpt_simple_soaph.py:271
↓ 1 callersMethod_initialize_blocked_parameters_state
(self)
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:618
↓ 1 callersMethod_initialize_state_lists
( self, block_list: tuple[Tensor, ...], kronecker_factors_unwrapped: list[_ShampooKron
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/shampoo_preconditioner_list.py:1261
↓ 1 callersMethod_instantiate_block_list
(self)
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/tests/preconditioner_list_test_utils.py:59
↓ 1 callersMethod_instantiate_distributor
(self)
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:548
↓ 1 callersMethod_instantiate_filtered_grads
(self)
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:790
↓ 1 callersMethod_instantiate_grafting
(self)
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:749
↓ 1 callersMethod_instantiate_iterate_averaging
(self)
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:829
↓ 1 callersMethod_instantiate_lr_tensors
Pre-allocate persistent lr tensors to avoid per-step pinned memory allocation.
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:776
↓ 1 callersMethod_instantiate_per_group_step
( self, shampoo_pt2_compile_config: ShampooPT2CompileConfig | None )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:894
↓ 1 callersMethod_instantiate_preconditioner_list
( self, **kwargs: object )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/tests/sgd_preconditioner_list_test.py:29
↓ 1 callersMethod_instantiate_preconditioner_list
( self, **kwargs: Any )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/tests/preconditioner_list_test_utils.py:62
↓ 1 callersMethod_instantiate_preconditioner_list
( self, **kwargs: Any )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/tests/spectral_descent_preconditioner_list_test.py:46
↓ 1 callersMethod_instantiate_preconditioner_list
( self, **kwargs: object )
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/preconditioner/tests/sign_descent_preconditioner_list_test.py:40
↓ 1 callersMethod_instantiate_shampoo_preconditioner_list
(self)
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:733
↓ 1 callersMethod_instantiate_steps
(self)
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:762
↓ 1 callersMethod_launch_gather
Launch async all_gather for a sharded parameter.
train_gpt.py:623
↓ 1 callersMethod_launch_reduce
Launch async reduce for a parameter based on its comms policy.
train_gpt.py:581
↓ 1 callersFunction_linear_ramp
(step: int, start_step: int, end_step: int)
records/track_3_optimization/results/20260525_aurora_ema_ref/train_gpt_simple_aurora_ema_ref.py:387
↓ 1 callersFunction_load_data_shard
(path)
records/track_2_medium/2024-12-31_Target350M/train_gpt.py:325
↓ 1 callersFunction_load_data_shard
(filename)
records/track_1_short/2024-12-04_ValueEmbed/train_gpt2.py:306
↓ 1 callersFunction_lr
(step, initial_lr, power_c, power=1.0)
records/track_3_optimization/results/20260520_rre_extrapolation_pr300_2925/train_gpt_simple_rre_pr300_2925.py:1455
↓ 1 callersFunction_lr
(step, initial_lr, power_c, power=1.0)
records/track_3_optimization/results/20260611_tailema_2720_submission/train_gpt_tailema_2720.py:996
↓ 1 callersFunction_lr
(step, initial_lr, power_c, power=1.0)
records/track_3_optimization/results/20260609_soap_f1_auxb2_clean/train_gpt_clean_SOTA.py:806
↓ 1 callersFunction_lr
(step, initial_lr, power_c, power=1.0)
records/track_3_optimization/results/20260529_tail_phase_readout_2850/train_gpt_tail_phase_readout_2850.py:2180
↓ 1 callersFunction_lr
(step, initial_lr, power_c, power=1.0)
records/track_3_optimization/results/20260619_cwd_rowfloor_tailema/train_gpt_cwd_SOTA.py:850
↓ 1 callersFunction_lr
(step, initial_lr, power_c, power=1.0)
records/track_3_optimization/results/20260525_aurora_ema_ref/train_gpt_simple_aurora_ema_ref.py:689
↓ 1 callersFunction_lr
(step, initial_lr, power_c, power=1.0)
records/track_3_optimization/results/20260611_tailema_2730_submission/train_gpt_tailema_2730.py:995
↓ 1 callersFunction_lr
(step, initial_lr, power_c, power=1.0)
records/track_3_optimization/results/20260611_tailema_2730_submission/ablation/combo_2740/train_gpt_combo_2740.py:972
↓ 1 callersFunction_lr_sprint_scale_at_step
(step)
records/track_3_optimization/results/20260520_rre_extrapolation_pr300_2925/train_gpt_simple_rre_pr300_2925.py:1461
↓ 1 callersMethod_mask_state_lists
( state_lists: dict[str, Any], group: dict[str, Any], shampoo_pt2_enabled: bool = Fals
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributed_shampoo.py:912
↓ 1 callersMethod_maybe_extrapolate
(self, params)
records/track_3_optimization/results/20260520_rre_extrapolation_pr300_2925/train_gpt_simple_rre_pr300_2925.py:1113
↓ 1 callersMethod_maybe_switch
(self)
train_gpt.py:1549
↓ 1 callersMethod_merge_and_block_gradients
Merges small dims and blocks gradients. NOTE: FSDP Distributor may modify this function. Returns: local_masked_blocked_g
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_distributor.py:238
↓ 1 callersMethod_merge_and_block_parameters
Merges small dimensions and blocks parameters, storing results as instance attributes. NOTE: FSDP may modify this function.
records/track_3_optimization/results/20260513_shampoo_1_4_power/distributed_shampoo/distributor/shampoo_distributor.py:221
↓ 1 callersFunction_muon_mu_at_step
(step, train_steps)
records/track_3_optimization/results/20260520_rre_extrapolation_pr300_2925/train_gpt_simple_rre_pr300_2925.py:1478
↓ 1 callersFunction_muon_mu_at_step
(step, train_steps)
records/track_3_optimization/results/20260611_tailema_2720_submission/train_gpt_tailema_2720.py:1011
↓ 1 callersFunction_muon_mu_at_step
(step, train_steps)
records/track_3_optimization/results/20260609_soap_f1_auxb2_clean/train_gpt_clean_SOTA.py:819
↓ 1 callersFunction_muon_mu_at_step
(step, train_steps)
records/track_3_optimization/results/20260529_tail_phase_readout_2850/train_gpt_tail_phase_readout_2850.py:2192
↓ 1 callersFunction_muon_mu_at_step
(step, train_steps)
records/track_3_optimization/results/20260619_cwd_rowfloor_tailema/train_gpt_cwd_SOTA.py:863
↓ 1 callersFunction_muon_mu_at_step
(step, train_steps)
records/track_3_optimization/results/20260520_tail_refinterp_2900/train_gpt_tail_refinterp_2900.py:1548
↓ 1 callersFunction_muon_mu_at_step
(step, train_steps)
records/track_3_optimization/results/20260525_aurora_ema_ref/train_gpt_simple_aurora_ema_ref.py:701
↓ 1 callersFunction_muon_mu_at_step
(step, train_steps)
records/track_3_optimization/results/20260611_tailema_2730_submission/train_gpt_tailema_2730.py:1010
↓ 1 callersFunction_muon_mu_at_step
(step, train_steps)
records/track_3_optimization/results/20260611_tailema_2730_submission/ablation/combo_2740/train_gpt_combo_2740.py:985
↓ 1 callersMethod_normuon_update
Apply NorMuon update to a parameter. Returns the updated p_slice.
train_gpt.py:862
← previousnext →501–600 of 2,320, ranked by callers