sk-spark

Discrete event simulator for Spark on Kubernetes

Description

sk-spark is part of the SimKube suite of tools, and can be used for analyzing disruption of Spark jobs running on top of Kubernetes. To build, run cargo build or make build (please ensure you've cloned the build submodule in this case by running git submodule update --init --recursive).

To run, provide a "plan" file in JSON format following the schema below:

{
    "cluster": {
        "nodeCapacities": [list of node capacities in terms of "number of executors per node"],
        "consolidateAfter": number of seconds before a node can be considered for consolidation,
    },
    "jobs": [
        {
            "launchTime": timestamp in seconds when the job should launch,
            "numExecutors": number of executors in the job,
            "numTasks": number of tasks for the job,
            "durationMean": mean task run time,
            "durationStdev": standard deviation for task run time,
            "outlierPercent": likelihood that a task is an "outlier"
            "maxRetries": number of retries for a task before the job is marked failed
        },
        ...
    ]
}

Each job in the plan will be scheduled onto the cluster at the specified launch time; its executors will be scheduled onto nodes in a round-robin fashion, and tasks will be assigned to executors in a first-come, first-serve manner. Each task has a randomly-generated runtime, normally distributed with a user-specified "outlier" percentage; an "outlier" task tasks 10x as long to complete.

sk-spark provides a basic model of a Kubernetes cluster; each node in the cluster has a number of "slots", indicating the number of executors that can fit on that node. New nodes will be launched when there are waiting executors, and nodes will be consolidated as space becomes available in the cluster. sk-spark mimics the consolidation behaviour of Karpenter. On each iteration, it looks at every node to determine if it is eligible for consolidation, specifically if the specified number of seconds since the last scheduling event have elapsed, and if all of the executor pods on a node can be rescheduled elsewhere.

If a node is selected for consolidation, executor pods are placed back in the "waiting" queue, and the tasks those executors were running are reset. If a given task is interrupted too many times (exceeding the maxRetries value), the job will be marked as "failed", otherwise the job is successful once all tasks have completed.