MCPcopy Create free account

hub / github.com/ArchieGertsman/spark-sched-sim / functions

Functions248 in github.com/ArchieGertsman/spark-sched-sim

↓ 7 callersMethodremove
(self, item)
spark_sched_sim/data_samplers/tpch.py:285
↓ 6 callersMethod__init__
( self, num_node_features: int, embed_dim: int, mlp_kwargs: dict[str, Any] )
schedulers/decima/scheduler.py:245
↓ 6 callersMethodadd
(self, item)
spark_sched_sim/data_samplers/tpch.py:276
↓ 6 callersMethodnum_committable_execs
(self)
spark_sched_sim/components/executor_tracker.py:105
↓ 6 callersMethodpop
(self)
spark_sched_sim/components/event.py:44
↓ 5 callersMethod_find_schedulable_stages
An stage is schedulable if it is ready (see `_is_stage_ready()`), it hasn't been selected in the current scheduling round, and its job
spark_sched_sim/spark_sched_sim.py:505
↓ 5 callersMethod_move_idle_executors
When an executor becomes idle, it may need to be moved somewhere. If it's idle at a stage, it might need to be moved to the job pool.
spark_sched_sim/spark_sched_sim.py:745
↓ 5 callersMethod_sample_task_duration
raises an exception if `executor_key` is not found in the durations from `wave`
spark_sched_sim/data_samplers/tpch.py:208
↓ 5 callersMethodclear
(self)
spark_sched_sim/data_samplers/tpch.py:282
↓ 5 callersMethodmove_executor_to_pool
( self, executor_id: int, new_pool_key: OptPoolKey, send: bool = False )
spark_sched_sim/components/executor_tracker.py:186
↓ 4 callersMethodis_at_job
(self, job_id: int)
spark_sched_sim/components/executor.py:31
↓ 4 callersMethodreset
(self)
spark_sched_sim/components/event.py:27
↓ 4 callersMethodstep
(self, action: dict)
spark_sched_sim/spark_sched_sim.py:188
↓ 3 callersMethod_move_executor_to_stage
(self, executor: Executor, stage: Stage)
spark_sched_sim/spark_sched_sim.py:799
↓ 3 callersMethod_observe
(self)
spark_sched_sim/spark_sched_sim.py:345
↓ 3 callersMethodadd
( self, obs: dict, wall_time: float, action: tuple, lgprob: float,
trainers/rollout_worker.py:27
↓ 3 callersMethodclose
(self)
spark_sched_sim/components/renderer.py:75
↓ 3 callersFunctioncounts_to_ptr
(x: Tensor)
schedulers/decima/utils.py:82
↓ 3 callersFunctionfind_stage
searches for a schedulable stage in a given job, prioritizing frontier stages
schedulers/heuristics/utils.py:17
↓ 3 callersFunctionjob_durations
(env)
spark_sched_sim/metrics.py:4
↓ 3 callersFunctionload
(filename=None)
cfg_loader.py:5
↓ 3 callersFunctionmake_scheduler
(agent_cfg)
schedulers/__init__.py:17
↓ 3 callersFunctionptr_to_counts
(ptr)
schedulers/decima/utils.py:78
↓ 3 callersMethodpush
(self, t: float, event: Event)
spark_sched_sim/components/event.py:34
↓ 3 callersMethodschedule
(self, obs: dict)
schedulers/scheduler.py:17
↓ 3 callersMethodtrain
trains the model on different job arrival sequences. For each job sequence: - multiple rollouts are collected in parallel, asynchronou
trainers/trainer.py:85
↓ 3 callersMethodupdate_executor_source
(self, pool_key: PoolKey)
spark_sched_sim/components/executor_tracker.py:125
↓ 2 callersMethod__init__
(self, env)
schedulers/decima/env_wrapper.py:13
↓ 2 callersMethod_commit_remaining_executors
There may be executors at the current source pool that weren't committed anywhere, e.g. because there were no more stages to schedule,
spark_sched_sim/spark_sched_sim.py:487
↓ 2 callersMethod_execute_next_task
starts work on another one of `stage`'s tasks, assuming there are still tasks remaining and the executor is local to the stage
spark_sched_sim/spark_sched_sim.py:584
↓ 2 callersMethod_fulfill_commitment
(self, executor_id: int, dst_pool_key: PoolKey)
spark_sched_sim/spark_sched_sim.py:699
↓ 2 callersMethod_get_executor_demand
a stage's executor demand is the number of executors that it can accept in addition to the executors currently working on, committed to,
spark_sched_sim/spark_sched_sim.py:566
↓ 2 callersMethod_get_idle_source_executors
(self, pool_key: PoolKey | None = None)
spark_sched_sim/spark_sched_sim.py:714
↓ 2 callersMethod_is_stage_saturated
a stage is saturated if it doesn't need any more executors.
spark_sched_sim/spark_sched_sim.py:580
↓ 2 callersMethod_preprocess_rollouts
( self, rollout_buffers: Iterable[RolloutBuffer] )
trainers/trainer.py:172
↓ 2 callersMethodadd_commitment
(self, num_executors: int, dst_pool_key: PoolKey)
spark_sched_sim/components/executor_tracker.py:146
↓ 2 callersMethodadd_history
should be called whenever this executor is released from a job
spark_sched_sim/components/executor.py:34
↓ 2 callersMethodavg_job_duration
(self)
spark_sched_sim/spark_sched_sim.py:244
↓ 2 callersMethodclear_executor_source
(self)
spark_sched_sim/components/executor_tracker.py:128
↓ 2 callersMethoddetach_executor
(self, executor: Executor)
spark_sched_sim/components/job.py:86
↓ 2 callersMethodevaluate_actions
( self, obsns: Iterable[dict], actions: Iterable[tuple] )
schedulers/scheduler.py:28
↓ 2 callersMethodexec_supply
(self, job_id: int)
spark_sched_sim/components/executor_tracker.py:122
↓ 2 callersMethodget_pool
(self, pool_key: OptPoolKey)
spark_sched_sim/components/executor_tracker.py:134
↓ 2 callersFunctionmake_trainer
(cfg)
trainers/__init__.py:7
↓ 2 callersMethodpool_size
(self, pool_key: OptPoolKey)
spark_sched_sim/components/executor_tracker.py:137
↓ 2 callersFunctionpreprocess_obs
(obs: dict[str, Any])
schedulers/heuristics/utils.py:5
↓ 2 callersFunctionrun_episode
(env_cfg, scheduler, seed=1234)
examples.py:84
↓ 2 callersMethodsource_job_id
(self)
spark_sched_sim/components/executor_tracker.py:99
↓ 2 callersMethodupdate_parameters
(self, loss: Tensor | None = None)
schedulers/scheduler.py:37
↓ 1 callersMethod__init__
(self, async_rollouts: bool = False)
trainers/rollout_worker.py:19
↓ 1 callersMethod_adjust_num_executors
truncates the numer of executor assigned to `stage` to the stage's demand, if it's larger
spark_sched_sim/spark_sched_sim.py:557
↓ 1 callersMethod_average
(self, ts_list, ys_list)
trainers/utils/baselines.py:20
↓ 1 callersMethod_build_node_features
( self, obs: dict[str, Any], commit_caps: ndarray )
schedulers/decima/env_wrapper.py:110
↓ 1 callersMethod_calc_differential_returns
(self, dt_list, rewards_list)
trainers/utils/returns_calculator.py:52
↓ 1 callersMethod_calc_discounted_returns
(self, dt_list, rewards_list)
trainers/utils/returns_calculator.py:67
↓ 1 callersMethod_capture_state
( self, i: int, avg_num_jobs: float, state_dict: dict, stats_list: Iterable[dict] )
trainers/trainer.py:244
↓ 1 callersMethod_check_dependencies
searches to see if all the dependencies of stage with id `stage_id` are satisfied.
spark_sched_sim/components/job.py:100
↓ 1 callersMethod_checkpoint
(self, i: int, best_state: dict)
trainers/trainer.py:256
↓ 1 callersMethod_cleanup
(self)
trainers/trainer.py:236
↓ 1 callersMethod_compute_jobtime
( self, wall_time_step: float, active_job_ids_step: list[int] )
spark_sched_sim/spark_sched_sim.py:847
↓ 1 callersMethod_compute_loss
CLIP loss
trainers/ppo.py:105
↓ 1 callersMethod_decrement_commitments
( self, src_pool_key: PoolKey, dst_pool_key: PoolKey )
spark_sched_sim/components/executor_tracker.py:238
↓ 1 callersMethod_download_tpch_dataset
(cls)
spark_sched_sim/data_samplers/tpch.py:109
↓ 1 callersMethod_draw_job_completion_markers
(self, canvas, job_completion_times, wall_time)
spark_sched_sim/components/renderer.py:120
↓ 1 callersMethod_draw_worker_histories
(self, canvas, worker_histories, wall_time)
spark_sched_sim/components/renderer.py:83
↓ 1 callersMethod_find_backup_stage
(self, executor: Executor)
spark_sched_sim/spark_sched_sim.py:821
↓ 1 callersMethod_find_new_frontier_stages
if ` stage` is completed, returns all of its successors whose other dependencies are also completed, if any exist.
spark_sched_sim/components/job.py:113
↓ 1 callersMethod_forward_no_mp
forward pass without any message passing. Needed whenever all the active jobs are almost complete and only have a single layer of node
schedulers/decima/scheduler.py:236
↓ 1 callersMethod_fulfill_commitments_from_source
(self)
spark_sched_sim/spark_sched_sim.py:730
↓ 1 callersMethod_get_exec_actions
(self, exec_mask: Tensor)
schedulers/decima/scheduler.py:379
↓ 1 callersMethod_get_source_stages
(self)
spark_sched_sim/components/job.py:108
↓ 1 callersMethod_handle_event
(self, event: Event)
spark_sched_sim/spark_sched_sim.py:317
↓ 1 callersMethod_handle_job_arrival
(self, job: Job)
spark_sched_sim/spark_sched_sim.py:428
↓ 1 callersMethod_handle_released_executor
called upon a task completion. if the executor has been commited to a next stage, then try assigning it there. Otherwise, if `stage` became
spark_sched_sim/spark_sched_sim.py:639
↓ 1 callersMethod_increment_commitments
(self, dst_pool_key: PoolKey, n: int)
spark_sched_sim/components/executor_tracker.py:224
↓ 1 callersMethod_init_executor_intervals
(self, exec_cap)
spark_sched_sim/data_samplers/tpch.py:237
↓ 1 callersMethod_init_frontier
returns a set containing all the stages which are source nodes in the dag, i.e. which have no dependencies
spark_sched_sim/components/job.py:93
↓ 1 callersMethod_is_stage_ready
a stage is ready if - it is unsaturated, and - all of its parent stages are saturated
spark_sched_sim/spark_sched_sim.py:542
↓ 1 callersMethod_load_initial_jobs
(self)
spark_sched_sim/spark_sched_sim.py:260
↓ 1 callersMethod_load_query
(cls, query_num, query_size)
spark_sched_sim/data_samplers/tpch.py:118
↓ 1 callersMethod_make_text_surfaces
( self, wall_time, avg_job_duration, num_active_jobs, num_jobs_completed, dy=20 )
spark_sched_sim/components/renderer.py:128
↓ 1 callersMethod_pre_process_task_duration
(cls, task_duration)
spark_sched_sim/data_samplers/tpch.py:135
↓ 1 callersMethod_process_job_completion
performs some bookkeeping when a job completes
spark_sched_sim/spark_sched_sim.py:682
↓ 1 callersMethod_process_stage_completion
performs some bookkeeping when a stage completes
spark_sched_sim/spark_sched_sim.py:676
↓ 1 callersMethod_render_frame
(self)
spark_sched_sim/spark_sched_sim.py:408
↓ 1 callersMethod_reset_biases
(self)
schedulers/decima/scheduler.py:66
↓ 1 callersMethod_reset_edge_links
(self)
spark_sched_sim/spark_sched_sim.py:249
↓ 1 callersMethod_resume_simulation
resumes the simulation until either there are new scheduling decisions to be made, or it's done.
spark_sched_sim/spark_sched_sim.py:320
↓ 1 callersMethod_rough_task_duration
(cls, task_duration_data)
spark_sched_sim/data_samplers/tpch.py:162
↓ 1 callersMethod_sample_executor_key
(self, data, num_local_executors)
spark_sched_sim/data_samplers/tpch.py:216
↓ 1 callersMethod_sample_job
(self, job_id, t_arrival)
spark_sched_sim/data_samplers/tpch.py:176
↓ 1 callersMethod_send_executor
sends a `executor` to `stage`, assuming that the executor is currently at a different job
spark_sched_sim/spark_sched_sim.py:617
↓ 1 callersMethod_setup
(self)
trainers/trainer.py:214
↓ 1 callersMethod_start_rollout_workers
(self)
trainers/trainer.py:264
↓ 1 callersMethod_take_action
(self, action: dict)
spark_sched_sim/spark_sched_sim.py:275
↓ 1 callersMethod_terminate_rollout_workers
(self)
trainers/trainer.py:298
↓ 1 callersMethod_train
(self, dataloader)
trainers/ppo.py:73
↓ 1 callersMethod_try_backup_schedule
If a executor arrives to a stage that no longer needs any executors, then greedily try to find a backup stage.
spark_sched_sim/spark_sched_sim.py:784
↓ 1 callersMethod_update_avg_num_jobs
(self, deltas_list, rewards_list)
trainers/utils/returns_calculator.py:78
↓ 1 callersMethod_update_executor_source
called upon a task completion. If any new stages were unlocked within this job upon the task completion, then start a new commitment r
spark_sched_sim/spark_sched_sim.py:662
next →1–100 of 248, ranked by callers