clumsy-cluster

A standalone discrete-event cluster simulator with:

• a native Python simulator API;
• a centralized Gymnasium API; and
• a two-agent PettingZoo Parallel API.

Architecture

Each concern has one direction of dependency:

flowchart LR
    PF[ProcessFlow<br/>machines, demand, arrivals] --> SIM[ClusterSimulator<br/>event-driven physics]
    OBS[SimulationObservation] --> ALLOC[Allocation decoders<br/>online + batch actions]
    ALLOC --> SIM
    SIM --> OBS
    GYM[Gymnasium adapter] --> ALLOC
    GYM --> SIM
    PZ[PettingZoo adapter] --> GYM

Loading

Concern	Module	Owns
Process input	cluster_sim.processes	Machine topology, online-demand samples, batch-process arrivals
Allocation agents	cluster_sim.allocation	Discrete action meanings and conversion to one allocation
Simulator	cluster_sim.simulator	Queueing, placement, CPU sharing, memory, time, and metrics
Gymnasium	cluster_sim.envs.gymnasium	One 18-choice joint action
PettingZoo	cluster_sim.envs.pettingzoo	Separate online and batch actions

The simulator core does not import Gymnasium or PettingZoo. Both APIs execute the same ClusterSimulator.

Install

From this directory:

python -m pip install -e '.[test]'
pytest -q

Python 3.11 or newer is required.

Define the process flow

ProcessFlow is the only input to the simulator. Online work is a per-machine CPU-demand time series. Batch work is a collection of BatchProcess objects.

from cluster_sim import (
    BatchProcess,
    DemandSample,
    Machine,
    ProcessFlow,
)

flow = ProcessFlow(
    machines=(
        Machine("machine-0", cpu=8, memory=32, online_memory=8),
    ),
    processes=(
        BatchProcess(
            "process-a",
            release=0,
            cpu=4,
            memory=8,
            duration=10,
            estimate=12,
        ),
        BatchProcess(
            "process-b",
            release=15,
            cpu=2,
            memory=4,
            duration=20,
            estimate=20,
        ),
    ),
    demand=(
        DemandSample(0, cpu=(2,)),
        DemandSample(30, cpu=(5,)),
    ),
    horizon=120,
    provenance="example",
)

The flow is immutable. It describes what enters the cluster and when; it does not allocate resources or execute processes.

Use the simulator directly

from cluster_sim import Allocation, ClusterSimulator, SimulatorConfig

simulator = ClusterSimulator(
    flow,
    SimulatorConfig(
        actuation_delay=0,
        warmup=0,
        borrowing=True,
    ),
)

simulator.submit(
    Allocation(
        online_quota=(3,),
        queue_order="fifo",
        admission_fraction=1.0,
    )
)
observation = simulator.advance(60)

print(simulator.status("process-a"))
print(simulator.report())

ClusterSimulator is responsible only for physics. It accepts an Allocation and advances to an absolute simulation time.

Use Gymnasium

Gymnasium combines the online and batch decisions into one Discrete(18) action.

from cluster_sim import synthetic_process_flow
from cluster_sim.envs import ClusterEnv

env = ClusterEnv(synthetic_process_flow(seed=7))
observation, info = env.reset(seed=7)

while True:
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        break

env.close()

The encoding is:

joint_action = online_action * 6 + batch_action
online_action, batch_action = divmod(joint_action, 6)

Use PettingZoo with two agents

PettingZoo exposes a simultaneous Parallel API. Every step requires one action from each live agent.

from cluster_sim import synthetic_process_flow
from cluster_sim.envs import ClusterParallelEnv

env = ClusterParallelEnv(synthetic_process_flow(seed=7))
observations, infos = env.reset(seed=7)

while env.agents:
    actions = {
        "online": online_policy(observations["online"]),
        "batch": batch_policy(observations["batch"]),
    }
    (
        observations,
        rewards,
        terminations,
        truncations,
        infos,
    ) = env.step(actions)

env.close()

Both agents receive the same reward. The online agent sees machine and summary state but not per-process queue rows. The batch agent sees the complete encoded observation. env.state() returns the unmasked global observation.

Agent actions

The allocation module converts actions; the environments do not contain allocation-policy logic.

Online action	Online quota
0	Current demand
1	Current demand + 10% machine capacity
2	Current demand + 20% machine capacity

Quota is capped at machine capacity.

Batch action	Queue order	CPU admission
0	FIFO	75%
1	FIFO	100%
2	Shortest estimate first	75%
3	Shortest estimate first	100%
4	Smallest dominant resource share first	75%
5	Smallest dominant resource share first	100%

To replace these mappings, call ClusterSimulator directly with custom Allocation objects or implement a different environment adapter.

Process lifecycle

stateDiagram-v2
    [*] --> Unreleased
    Unreleased --> Queued: release time
    Queued --> Running: CPU and memory fit
    Running --> Running: demand or quota changes its CPU rate
    Running --> Finished: remaining work reaches zero
    Queued --> Censored: horizon
    Running --> Censored: horizon

Loading

For each process:

1. At release, it enters the queue.
2. The selected queue order determines when it is considered.
3. Deterministic best-fit placement selects a machine with enough admitted CPU and memory.
4. Its required work is cpu × duration core-seconds.
5. If running processes request more CPU than the current batch budget, they slow down by the same fraction. Required work does not change.
6. Memory remains allocated until completion.
7. Completion releases CPU and memory immediately, then queued work is considered again.

Running processes are not migrated or evicted. An aging guard prevents younger processes from indefinitely blocking the oldest feasible process.

Online demand is not a queued process. Without borrowing, batch capacity is machine capacity minus online quota. With borrowing enabled, batch work may use idle online capacity, but a rise in online demand immediately reduces its CPU rate.

Step and reward semantics

One environment step advances by SimulatorConfig.control_interval, 60 seconds by default. Internally, the simulator jumps between exact event times: releases, demand samples, delayed allocations, completions, and the control boundary.

The shared reward is the negative cost added during that step. Cost combines:

• online CPU shortfall;
• unfinished batch processes; and
• quota-change penalty.

The horizon is a truncation, not a natural termination. Reports include unfinished count, remaining work, and restricted flow time so policies are not rewarded for leaving difficult processes incomplete.

Package layout

src/cluster_sim/
├── allocation.py
├── models.py
├── processes.py
├── simulator.py
└── envs/
    ├── _encoding.py
    ├── gymnasium.py
    └── pettingzoo.py