Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/ArchieGertsman/spark-sched-sim
/ functions
Functions
248 in github.com/ArchieGertsman/spark-sched-sim
⨍
Functions
248
◇
Types & classes
41
↓ 7 callers
Method
remove
(self, item)
spark_sched_sim/data_samplers/tpch.py:285
↓ 6 callers
Method
__init__
( self, num_node_features: int, embed_dim: int, mlp_kwargs: dict[str, Any] )
schedulers/decima/scheduler.py:245
↓ 6 callers
Method
add
(self, item)
spark_sched_sim/data_samplers/tpch.py:276
↓ 6 callers
Method
num_committable_execs
(self)
spark_sched_sim/components/executor_tracker.py:105
↓ 6 callers
Method
pop
(self)
spark_sched_sim/components/event.py:44
↓ 5 callers
Method
_find_schedulable_stages
An stage is schedulable if it is ready (see `_is_stage_ready()`), it hasn't been selected in the current scheduling round, and its job
spark_sched_sim/spark_sched_sim.py:505
↓ 5 callers
Method
_move_idle_executors
When an executor becomes idle, it may need to be moved somewhere. If it's idle at a stage, it might need to be moved to the job pool.
spark_sched_sim/spark_sched_sim.py:745
↓ 5 callers
Method
_sample_task_duration
raises an exception if `executor_key` is not found in the durations from `wave`
spark_sched_sim/data_samplers/tpch.py:208
↓ 5 callers
Method
clear
(self)
spark_sched_sim/data_samplers/tpch.py:282
↓ 5 callers
Method
move_executor_to_pool
( self, executor_id: int, new_pool_key: OptPoolKey, send: bool = False )
spark_sched_sim/components/executor_tracker.py:186
↓ 4 callers
Method
is_at_job
(self, job_id: int)
spark_sched_sim/components/executor.py:31
↓ 4 callers
Method
reset
(self)
spark_sched_sim/components/event.py:27
↓ 4 callers
Method
step
(self, action: dict)
spark_sched_sim/spark_sched_sim.py:188
↓ 3 callers
Method
_move_executor_to_stage
(self, executor: Executor, stage: Stage)
spark_sched_sim/spark_sched_sim.py:799
↓ 3 callers
Method
_observe
(self)
spark_sched_sim/spark_sched_sim.py:345
↓ 3 callers
Method
add
( self, obs: dict, wall_time: float, action: tuple, lgprob: float,
trainers/rollout_worker.py:27
↓ 3 callers
Method
close
(self)
spark_sched_sim/components/renderer.py:75
↓ 3 callers
Function
counts_to_ptr
(x: Tensor)
schedulers/decima/utils.py:82
↓ 3 callers
Function
find_stage
searches for a schedulable stage in a given job, prioritizing frontier stages
schedulers/heuristics/utils.py:17
↓ 3 callers
Function
job_durations
(env)
spark_sched_sim/metrics.py:4
↓ 3 callers
Function
load
(filename=None)
cfg_loader.py:5
↓ 3 callers
Function
make_scheduler
(agent_cfg)
schedulers/__init__.py:17
↓ 3 callers
Function
ptr_to_counts
(ptr)
schedulers/decima/utils.py:78
↓ 3 callers
Method
push
(self, t: float, event: Event)
spark_sched_sim/components/event.py:34
↓ 3 callers
Method
schedule
(self, obs: dict)
schedulers/scheduler.py:17
↓ 3 callers
Method
train
trains the model on different job arrival sequences. For each job sequence: - multiple rollouts are collected in parallel, asynchronou
trainers/trainer.py:85
↓ 3 callers
Method
update_executor_source
(self, pool_key: PoolKey)
spark_sched_sim/components/executor_tracker.py:125
↓ 2 callers
Method
__init__
(self, env)
schedulers/decima/env_wrapper.py:13
↓ 2 callers
Method
_commit_remaining_executors
There may be executors at the current source pool that weren't committed anywhere, e.g. because there were no more stages to schedule,
spark_sched_sim/spark_sched_sim.py:487
↓ 2 callers
Method
_execute_next_task
starts work on another one of `stage`'s tasks, assuming there are still tasks remaining and the executor is local to the stage
spark_sched_sim/spark_sched_sim.py:584
↓ 2 callers
Method
_fulfill_commitment
(self, executor_id: int, dst_pool_key: PoolKey)
spark_sched_sim/spark_sched_sim.py:699
↓ 2 callers
Method
_get_executor_demand
a stage's executor demand is the number of executors that it can accept in addition to the executors currently working on, committed to,
spark_sched_sim/spark_sched_sim.py:566
↓ 2 callers
Method
_get_idle_source_executors
(self, pool_key: PoolKey | None = None)
spark_sched_sim/spark_sched_sim.py:714
↓ 2 callers
Method
_is_stage_saturated
a stage is saturated if it doesn't need any more executors.
spark_sched_sim/spark_sched_sim.py:580
↓ 2 callers
Method
_preprocess_rollouts
( self, rollout_buffers: Iterable[RolloutBuffer] )
trainers/trainer.py:172
↓ 2 callers
Method
add_commitment
(self, num_executors: int, dst_pool_key: PoolKey)
spark_sched_sim/components/executor_tracker.py:146
↓ 2 callers
Method
add_history
should be called whenever this executor is released from a job
spark_sched_sim/components/executor.py:34
↓ 2 callers
Method
avg_job_duration
(self)
spark_sched_sim/spark_sched_sim.py:244
↓ 2 callers
Method
clear_executor_source
(self)
spark_sched_sim/components/executor_tracker.py:128
↓ 2 callers
Method
detach_executor
(self, executor: Executor)
spark_sched_sim/components/job.py:86
↓ 2 callers
Method
evaluate_actions
( self, obsns: Iterable[dict], actions: Iterable[tuple] )
schedulers/scheduler.py:28
↓ 2 callers
Method
exec_supply
(self, job_id: int)
spark_sched_sim/components/executor_tracker.py:122
↓ 2 callers
Method
get_pool
(self, pool_key: OptPoolKey)
spark_sched_sim/components/executor_tracker.py:134
↓ 2 callers
Function
make_trainer
(cfg)
trainers/__init__.py:7
↓ 2 callers
Method
pool_size
(self, pool_key: OptPoolKey)
spark_sched_sim/components/executor_tracker.py:137
↓ 2 callers
Function
preprocess_obs
(obs: dict[str, Any])
schedulers/heuristics/utils.py:5
↓ 2 callers
Function
run_episode
(env_cfg, scheduler, seed=1234)
examples.py:84
↓ 2 callers
Method
source_job_id
(self)
spark_sched_sim/components/executor_tracker.py:99
↓ 2 callers
Method
update_parameters
(self, loss: Tensor | None = None)
schedulers/scheduler.py:37
↓ 1 callers
Method
__init__
(self, async_rollouts: bool = False)
trainers/rollout_worker.py:19
↓ 1 callers
Method
_adjust_num_executors
truncates the numer of executor assigned to `stage` to the stage's demand, if it's larger
spark_sched_sim/spark_sched_sim.py:557
↓ 1 callers
Method
_average
(self, ts_list, ys_list)
trainers/utils/baselines.py:20
↓ 1 callers
Method
_build_node_features
( self, obs: dict[str, Any], commit_caps: ndarray )
schedulers/decima/env_wrapper.py:110
↓ 1 callers
Method
_calc_differential_returns
(self, dt_list, rewards_list)
trainers/utils/returns_calculator.py:52
↓ 1 callers
Method
_calc_discounted_returns
(self, dt_list, rewards_list)
trainers/utils/returns_calculator.py:67
↓ 1 callers
Method
_capture_state
( self, i: int, avg_num_jobs: float, state_dict: dict, stats_list: Iterable[dict] )
trainers/trainer.py:244
↓ 1 callers
Method
_check_dependencies
searches to see if all the dependencies of stage with id `stage_id` are satisfied.
spark_sched_sim/components/job.py:100
↓ 1 callers
Method
_checkpoint
(self, i: int, best_state: dict)
trainers/trainer.py:256
↓ 1 callers
Method
_cleanup
(self)
trainers/trainer.py:236
↓ 1 callers
Method
_compute_jobtime
( self, wall_time_step: float, active_job_ids_step: list[int] )
spark_sched_sim/spark_sched_sim.py:847
↓ 1 callers
Method
_compute_loss
CLIP loss
trainers/ppo.py:105
↓ 1 callers
Method
_decrement_commitments
( self, src_pool_key: PoolKey, dst_pool_key: PoolKey )
spark_sched_sim/components/executor_tracker.py:238
↓ 1 callers
Method
_download_tpch_dataset
(cls)
spark_sched_sim/data_samplers/tpch.py:109
↓ 1 callers
Method
_draw_job_completion_markers
(self, canvas, job_completion_times, wall_time)
spark_sched_sim/components/renderer.py:120
↓ 1 callers
Method
_draw_worker_histories
(self, canvas, worker_histories, wall_time)
spark_sched_sim/components/renderer.py:83
↓ 1 callers
Method
_find_backup_stage
(self, executor: Executor)
spark_sched_sim/spark_sched_sim.py:821
↓ 1 callers
Method
_find_new_frontier_stages
if ` stage` is completed, returns all of its successors whose other dependencies are also completed, if any exist.
spark_sched_sim/components/job.py:113
↓ 1 callers
Method
_forward_no_mp
forward pass without any message passing. Needed whenever all the active jobs are almost complete and only have a single layer of node
schedulers/decima/scheduler.py:236
↓ 1 callers
Method
_fulfill_commitments_from_source
(self)
spark_sched_sim/spark_sched_sim.py:730
↓ 1 callers
Method
_get_exec_actions
(self, exec_mask: Tensor)
schedulers/decima/scheduler.py:379
↓ 1 callers
Method
_get_source_stages
(self)
spark_sched_sim/components/job.py:108
↓ 1 callers
Method
_handle_event
(self, event: Event)
spark_sched_sim/spark_sched_sim.py:317
↓ 1 callers
Method
_handle_job_arrival
(self, job: Job)
spark_sched_sim/spark_sched_sim.py:428
↓ 1 callers
Method
_handle_released_executor
called upon a task completion. if the executor has been commited to a next stage, then try assigning it there. Otherwise, if `stage` became
spark_sched_sim/spark_sched_sim.py:639
↓ 1 callers
Method
_increment_commitments
(self, dst_pool_key: PoolKey, n: int)
spark_sched_sim/components/executor_tracker.py:224
↓ 1 callers
Method
_init_executor_intervals
(self, exec_cap)
spark_sched_sim/data_samplers/tpch.py:237
↓ 1 callers
Method
_init_frontier
returns a set containing all the stages which are source nodes in the dag, i.e. which have no dependencies
spark_sched_sim/components/job.py:93
↓ 1 callers
Method
_is_stage_ready
a stage is ready if - it is unsaturated, and - all of its parent stages are saturated
spark_sched_sim/spark_sched_sim.py:542
↓ 1 callers
Method
_load_initial_jobs
(self)
spark_sched_sim/spark_sched_sim.py:260
↓ 1 callers
Method
_load_query
(cls, query_num, query_size)
spark_sched_sim/data_samplers/tpch.py:118
↓ 1 callers
Method
_make_text_surfaces
( self, wall_time, avg_job_duration, num_active_jobs, num_jobs_completed, dy=20 )
spark_sched_sim/components/renderer.py:128
↓ 1 callers
Method
_pre_process_task_duration
(cls, task_duration)
spark_sched_sim/data_samplers/tpch.py:135
↓ 1 callers
Method
_process_job_completion
performs some bookkeeping when a job completes
spark_sched_sim/spark_sched_sim.py:682
↓ 1 callers
Method
_process_stage_completion
performs some bookkeeping when a stage completes
spark_sched_sim/spark_sched_sim.py:676
↓ 1 callers
Method
_render_frame
(self)
spark_sched_sim/spark_sched_sim.py:408
↓ 1 callers
Method
_reset_biases
(self)
schedulers/decima/scheduler.py:66
↓ 1 callers
Method
_reset_edge_links
(self)
spark_sched_sim/spark_sched_sim.py:249
↓ 1 callers
Method
_resume_simulation
resumes the simulation until either there are new scheduling decisions to be made, or it's done.
spark_sched_sim/spark_sched_sim.py:320
↓ 1 callers
Method
_rough_task_duration
(cls, task_duration_data)
spark_sched_sim/data_samplers/tpch.py:162
↓ 1 callers
Method
_sample_executor_key
(self, data, num_local_executors)
spark_sched_sim/data_samplers/tpch.py:216
↓ 1 callers
Method
_sample_job
(self, job_id, t_arrival)
spark_sched_sim/data_samplers/tpch.py:176
↓ 1 callers
Method
_send_executor
sends a `executor` to `stage`, assuming that the executor is currently at a different job
spark_sched_sim/spark_sched_sim.py:617
↓ 1 callers
Method
_setup
(self)
trainers/trainer.py:214
↓ 1 callers
Method
_start_rollout_workers
(self)
trainers/trainer.py:264
↓ 1 callers
Method
_take_action
(self, action: dict)
spark_sched_sim/spark_sched_sim.py:275
↓ 1 callers
Method
_terminate_rollout_workers
(self)
trainers/trainer.py:298
↓ 1 callers
Method
_train
(self, dataloader)
trainers/ppo.py:73
↓ 1 callers
Method
_try_backup_schedule
If a executor arrives to a stage that no longer needs any executors, then greedily try to find a backup stage.
spark_sched_sim/spark_sched_sim.py:784
↓ 1 callers
Method
_update_avg_num_jobs
(self, deltas_list, rewards_list)
trainers/utils/returns_calculator.py:78
↓ 1 callers
Method
_update_executor_source
called upon a task completion. If any new stages were unlocked within this job upon the task completion, then start a new commitment r
spark_sched_sim/spark_sched_sim.py:662
next →
1–100 of 248, ranked by callers