Discussion: evolve the pack for group RayService, user RayClusters, RayJobs and a Ray gateway #34

Definitions

RayService: Runs a long-lived Ray cluster for serving models and manages safe updates to its Ray Serve applications.

RayCluster: Runs a standalone Ray cluster for interactive or shared compute.

RayJob: Unlike a RayCluster, it is not primarily a cluster that a user connects to interactively. It is used for durable UI or API submissions and can create and clean up a temporary Ray cluster for the job.

Current topology

flowchart TB
    UI["Deployed UI and API"]
    NotebookA["User A JupyterLab"]
    NotebookB["User B JupyterLab"]
    Models["Model callers"]
    Shared["One shared RayService<br/>and managed RayCluster"]

    UI --> Shared
    NotebookA --> Shared
    NotebookB --> Shared
    Models --> Shared

The current pack provides one shared Ray environment for serving, UI compute and every notebook. There is no group separation, no user-owned RayCluster and no ephemeral RayJob cluster.

Proposed topology

Single group

flowchart TB
    Gateway["One platform Ray gateway<br/>authentication, authorization, routing and lifecycle"]
    UI["User through the shared UI"]
    NotebookA["User A JupyterLab"]
    NotebookB["User B JupyterLab"]

    subgraph Group["Trusted group's Ray namespace"]
        Serve["One shared RayService<br/>group model serving"]
        Job["RayJob<br/>ephemeral cluster"]
        ClusterA["User A RayCluster<br/>Ray Client endpoint"]
        ClusterB["User B RayCluster<br/>Ray Client endpoint"]
    end

    UI --> Gateway
    NotebookA --> Gateway
    NotebookB --> Gateway
    Gateway -->|authorized model requests| Serve
    Gateway -->|create and manage| Job
    Gateway -->|create and manage| ClusterA
    Gateway -->|create and manage| ClusterB
    NotebookA -.->|authorized Ray Client connection| ClusterA
    NotebookB -.->|authorized Ray Client connection| ClusterB

Why both RayJob and RayCluster?

A RayJob runs a complete command in a separate process. A notebook cannot use it to keep live Python objects, ObjectRef values or actor handles between cells. A RayCluster with Ray Client keeps the notebook connected, so users can submit work and reuse state interactively.

Multiple groups

flowchart TB
    IdP["OIDC identity and group membership"] --> Gateway["One platform Ray gateway"]

    UsersA["Group A users"] --> Gateway
    UsersB["Group B users"] --> Gateway

    subgraph GroupA["Group A namespace"]
        ServeA["RayService A<br/>models for Group A"]
        ClusterA["User RayClusters"]
        JobA["Ephemeral RayJobs"]
    end

    subgraph GroupB["Group B namespace"]
        ServeB["RayService B<br/>models for Group B"]
        ClusterB["User RayClusters"]
        JobB["Ephemeral RayJobs"]
    end

    Gateway -->|authorized Group A model requests| ServeA
    Gateway -->|authorized Group B model requests| ServeB
    Gateway -->|create and manage| ClusterA
    Gateway -->|create and manage| JobA
    Gateway -->|create and manage| ClusterB
    Gateway -->|create and manage| JobB

    Boundary["Namespaces, quotas and NetworkPolicy<br/>block cross-group access"]
    Boundary -.-> GroupA
    Boundary -.-> GroupB

    Nodes["Shared Kubernetes node pools"]
    ServeA -.-> Nodes
    ClusterA -.-> Nodes
    JobA -.-> Nodes
    ServeB -.-> Nodes
    ClusterB -.-> Nodes
    JobB -.-> Nodes

There is one Ray gateway for the platform, not one gateway per group. It derives the group from the authenticated identity, routes model requests to that group's RayService, and creates lifecycle resources inside the correct group namespace. Groups have separate RayServices, RayClusters, RayJobs, quotas and network rules, but their pods may still run on the same Kubernetes nodes.

The group is the boundary for model serving, permissions and quotas. Interactive compute is isolated further by giving each user or notebook session its own Ray cluster.

Desired user experience

We want the Ray pack to support these simple workflows:

• Each trusted group has one shared RayService for the models used by that group.
• Every model request is authenticated, and the caller must belong to the group that owns the RayService.
• Direct access to the internal Ray Serve Service is blocked so callers cannot bypass group authorization.
• A group's serving environment is separate from notebook work and UI jobs, so user compute cannot shut it down or consume its Ray resources.
• UI requests always create a RayJob. The job receives an ephemeral Ray cluster and that cluster is removed after the job finishes.
• A JupyterLab user can request their own RayCluster from a gateway and receive a compatible Ray Client address.
• Different users do not share a Ray head, workers, actors or object store unless they explicitly choose a shared cluster in the future.
• Users can list and stop their own clusters. Idle clusters are cleaned up safely.
• Group administrators can control which profiles, images, CPU, memory, GPU and maximum worker counts are allowed.
• Users choose approved options rather than submitting arbitrary Kubernetes manifests or service accounts.
• Users cannot connect to another group's Ray Client, dashboard, Jobs API or GCS endpoint.
• The gateway records ownership and can recover the state of existing RayCluster and RayJob resources after restarting.

This gives us three distinct KubeRay lifecycle patterns:

Authentication and group access

The platform gateway authenticates every user and uses group membership to decide what they can access. Users can call models, submit jobs, create clusters, and view resources only within their groups. Internal Ray endpoints are not exposed directly.

The dashboard for a shared group RayService is an operational interface, so it is available only to platform administrators and approved group operators. Ordinary users call group models through the deployed application UI. If users need model health or status, expose a limited group status view rather than the full Ray dashboard.

Users may access the dashboard for their own RayCluster because they own the workloads running there.

The following section was removed after discussion as it was premature.

## Evolving the current pack

The current repository can evolve into a more general Ray platform pack rather than creating unrelated implementations. A future name such as nebari-ray-pack would better reflect the wider scope, but existing chart and resource names should remain compatible during migration.

A possible structure is:

<PROJECT_LAYOUT_REMOVED>

RayService, RayCluster and RayJob all need many of the same cluster settings. The pack should define those settings once and reuse them, with different profiles for serving, interactive CPU, interactive GPU and UI jobs.

The platform part of the pack would provide the shared Ray services and one gateway. The group part would create the group's model-serving environment and access boundaries. When a user asks for notebook compute or submits a UI job, the gateway would create the right Ray resource from an approved profile and clean it up when it is no longer needed.

KubeRay would continue to manage the Ray resources underneath. This approach lets the current pack grow to support groups and user-owned compute without creating separate solutions that can drift apart.