MCPcopy Create free account
hub / github.com/apify/crawlee-python / SystemStatus

Class SystemStatus

src/crawlee/_autoscaling/system_status.py:21–201  ·  view source on GitHub ↗

Provides a simple interface for evaluating system resource usage from snapshots collected by `Snapshotter`. This class aggregates and interprets snapshots from a Snapshotter instance to evaluate the current and historical status of system resources like CPU, memory, event loop, and client A

Source from the content-addressed store, hash-verified

19
20@docs_group('Autoscaling')
21class SystemStatus:
22 """Provides a simple interface for evaluating system resource usage from snapshots collected by `Snapshotter`.
23
24 This class aggregates and interprets snapshots from a Snapshotter instance to evaluate the current and historical
25 status of system resources like CPU, memory, event loop, and client API usage. It exposes two methods
26 `get_current_system_info` and `get_historical_system_info`. The system information is computed using a weighted
27 average of overloaded messages in the snapshots, with the weights being the time intervals between the snapshots.
28 Each resource is computed separately, and the system is considered as overloaded whenever at least one resource
29 is overloaded.
30
31 `get_current_system_info` returns a `SystemInfo` data structure that represents the current status
32 of the system. The length of the current timeframe in seconds is configurable by the `max_snapshot_age` option
33 and represents the max age of snapshots to be considered for the computation.
34
35 `SystemStatus.get_historical_system_info` returns a `SystemInfo` that represents the long-term status of the system.
36 It considers the full snapshot history available in the `Snapshotter` instance.
37 """
38
39 def __init__(
40 self,
41 snapshotter: Snapshotter,
42 *,
43 max_snapshot_age: timedelta = timedelta(seconds=5),
44 cpu_overload_threshold: float = 0.4,
45 memory_overload_threshold: float = 0.2,
46 event_loop_overload_threshold: float = 0.6,
47 client_overload_threshold: float = 0.3,
48 ) -> None:
49 """Initialize a new instance.
50
51 Args:
52 snapshotter: The `Snapshotter` instance to be queried for `SystemStatus`.
53 max_snapshot_age: Defines max age of snapshots used in the `SystemStatus.get_current_system_info`
54 measurement.
55 cpu_overload_threshold: Sets the threshold of overloaded snapshots in the CPU sample.
56 If the sample exceeds this threshold, the system will be considered overloaded.
57 memory_overload_threshold: Sets the threshold of overloaded snapshots in the memory sample.
58 If the sample exceeds this threshold, the system will be considered overloaded.
59 event_loop_overload_threshold: Sets the threshold of overloaded snapshots in the event loop sample.
60 If the sample exceeds this threshold, the system will be considered overloaded.
61 client_overload_threshold: Sets the threshold of overloaded snapshots in the Client sample.
62 If the sample exceeds this threshold, the system will be considered overloaded.
63 """
64 self._snapshotter = snapshotter
65 self._max_snapshot_age = max_snapshot_age
66 self._cpu_overload_threshold = cpu_overload_threshold
67 self._memory_overload_threshold = memory_overload_threshold
68 self._event_loop_overload_threshold = event_loop_overload_threshold
69 self._client_overload_threshold = client_overload_threshold
70
71 def get_current_system_info(self) -> SystemInfo:
72 """Retrieve and evaluates the current status of system resources.
73
74 Considers snapshots within the `_max_snapshot_age` timeframe and determines if the system is currently
75 overloaded based on predefined thresholds for each resource type.
76
77 Returns:
78 An object representing the current system status.

Callers 7

__init__Method · 0.90
test_cpu_is_overloadedFunction · 0.90
test_get_system_infoFunction · 0.90
test_client_overloadedFunction · 0.90

Calls

no outgoing calls

Tested by 6

test_cpu_is_overloadedFunction · 0.72
test_get_system_infoFunction · 0.72
test_client_overloadedFunction · 0.72