Support for auto Statefulsets#856
Open
filippomc wants to merge 6 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds opt-in support for rendering Cloud Harness “auto” workloads as Kubernetes StatefulSets (instead of Deployments), including per-replica volume provisioning, an optional legacy-PVC migration mechanism, and ingress routing for “write” URIs to a leader (pod 0) via a -rw service. It updates the configuration models/OpenAPI/docs accordingly and extends tests and the samples app to exercise the new behavior.
Changes:
- Add
deployment.statefulset(app) anddatabase.statefulset(auto DB) options with StatefulSet rendering and per-replicavolumeClaimTemplates. - Add legacy PVC migration support via a rendered Job + initContainer gate (only while legacy PVC exists) and create a leader
*-rwService for StatefulSets. - Add
methodstouri_role_mappingand use it for ingress leader-routing behavior; update models/OpenAPI/docs and add tests/sample endpoint.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/deployment-cli-tools/tests/test_helm.py | Adds Helm-render tests covering StatefulSet option, leader service, and routing behavior. |
| libraries/models/docs/UriRoleMappingConfig.md | Documents new methods field semantics and leader-routing behavior. |
| libraries/models/docs/DeploymentAutoArtifactConfig.md | Documents new statefulset flag for app deployments. |
| libraries/models/docs/DatabaseDeploymentConfig.md | Documents new statefulset flag for auto-generated databases. |
| libraries/models/cloudharness_model/models/uri_role_mapping_config.py | Adds methods to the generated model and dict conversion. |
| libraries/models/cloudharness_model/models/deployment_auto_artifact_config.py | Adds statefulset to the generated deployment model and dict conversion. |
| libraries/models/cloudharness_model/models/database_deployment_config.py | Adds statefulset to the generated database model and dict conversion. |
| libraries/models/api/openapi.yaml | Extends OpenAPI schema with statefulset and methods fields and descriptions. |
| docs/model/UriRoleMappingConfig.md | Generated docs updated for methods. |
| docs/model/DeploymentAutoArtifactConfig.md | Generated docs updated for statefulset. |
| docs/model/DatabaseDeploymentConfig.md | Generated docs updated for statefulset. |
| docs/applications/volumes.md | Adds StatefulSet volume behavior, migration procedure, and leader-routing guidance. |
| docs/applications/databases.md | Documents database StatefulSet option and migration note. |
| deployment-configuration/value-template.yaml | Updates template comments and adds statefulset flags for deployment/database. |
| deployment-configuration/helm/templates/ingress.yaml | Adds ingress routing to *-rw service for write-method URI mappings (with secured-app guardrails). |
| deployment-configuration/helm/templates/auto-volumes.yaml | Adjusts PVC rendering rules for StatefulSet legacy-PVC migration behavior. |
| deployment-configuration/helm/templates/auto-volume-migration.yaml | Introduces migration Job + RBAC for copying legacy PVC data into StatefulSet volumes. |
| deployment-configuration/helm/templates/auto-services.yaml | Adds *-rw leader service that selects pod 0 for StatefulSets. |
| deployment-configuration/helm/templates/auto-deployments.yaml | Adds StatefulSet rendering, volumeClaimTemplates, and optional migration initContainer + resources. |
| deployment-configuration/helm/templates/auto-database.yaml | Adds database StatefulSet rendering, optional migration gate, and volumeClaimTemplates. |
| applications/samples/deploy/values.yaml | Updates sample deployment to demonstrate StatefulSet and leader-routed write endpoint. |
| applications/samples/deploy/values-statefulset.yaml | Adds a minimal StatefulSet sample values file. |
| applications/samples/backend/samples/test/test_sample.py | Adds unit test for the new write-file endpoint behavior. |
| applications/samples/backend/samples/controllers/test_controller.py | Adds write_file endpoint used to demonstrate single-writer/leader patterns. |
| applications/samples/api/openapi.yaml | Adds /write-file API definition and description for leader routing. |
Comment on lines
+215
to
+218
| {{- end }} | ||
| {{- if $legacyClaim }} | ||
| {{- include "deploy_utils.volumeMigration" (dict "root" .root "name" .app.harness.deployment.name "pvc" $volume.name) }} | ||
| {{- end }} |
Comment on lines
+144
to
+147
| {{- end }} | ||
| {{- if $legacyClaim }} | ||
| {{- include "deploy_utils.volumeMigration" (dict "root" .root "name" .app.harness.database.name "pvc" .app.harness.database.name) }} | ||
| {{- end }} |
Comment on lines
+51
to
+56
| try: | ||
| from cloudharness.applications import get_current_configuration | ||
| mountpath = get_current_configuration().harness.deployment.volume.mountpath | ||
| except Exception: | ||
| # not running inside a cloudharness deployment (e.g. unit tests) | ||
| mountpath = "/tmp/myvolume" |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
deployment-configuration/helm/templates/ingress.yaml:47
- This write-routing block always generates ingress paths to
<service>-rwwhendeployment.statefulsetis true, but the chart only creates that leader Service whenharness.service.autois enabled. Add the sameservice.autoguard here to avoid producing an ingress that points to a missing Service.
{{- /* uri_role_mapping entries declaring a write method are routed to the leader (pod 0)
service. Skipped for gatekeeper-secured apps, as routing straight to the service would
bypass authentication: those must forward writes at the application level. */}}
{{- if and $app.harness.deployment.statefulset $app.harness.uri_role_mapping (not (and $app.harness.secured $secured_gatekeepers)) (eq $app.harness.gateway.pathType "Prefix") }}
{{- range $mapping := $app.harness.uri_role_mapping }}
{{- if and $mapping.methods (or (has "POST" $mapping.methods) (has "PUT" $mapping.methods) (has "PATCH" $mapping.methods) (has "DELETE" $mapping.methods)) }}
applications/samples/backend/samples/controllers/test_controller.py:55
- Catching
Exceptionhere can mask real configuration/runtime errors in a deployed environment and silently fall back to writing into/tmp/myvolume(ephemeral container FS). Prefer a deterministic check for “running in CloudHarness” (e.g.,CH_CURRENT_APP_NAMEenv var) and only use the/tmpfallback in that case.
try:
from cloudharness.applications import get_current_configuration
mountpath = get_current_configuration().harness.deployment.volume.mountpath
except Exception:
# not running inside a cloudharness deployment (e.g. unit tests)
Comment on lines
+12
to
+15
| {{- /* White-listed uris bypass the gatekeeper; the ones declaring a write method are | ||
| routed to the leader (pod 0) service of a statefulset deployment. */}} | ||
| {{- $isWrite := and $app.harness.deployment.statefulset $mapping.methods (or (has "POST" $mapping.methods) (has "PUT" $mapping.methods) (has "PATCH" $mapping.methods) (has "DELETE" $mapping.methods)) }} | ||
| {{- $backendService := ternary (printf "%s-rw" $app.harness.service.name) $app.harness.service.name (not (not $isWrite)) }} |
Resolve auto-deployments.yaml: combine statefulset support (HEAD) with the old-helm-safe Recreate strategy and podAffinity guards from develop, via a nested-if $rwoVolume boolean so hasKey never runs on a nil volume. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
applications/samples/backend/samples/controllers/test_controller.py:55
- Catching a blanket
Exceptionhere can hide real configuration errors (e.g., misconfigured volume) and makes debugging in-cluster harder.get_current_configuration()raisesConfigurationCallExceptionwhen not running in a deployment; catching that (and optionally ImportError) keeps the unit-test fallback while surfacing unexpected failures.
try:
from cloudharness.applications import get_current_configuration
mountpath = get_current_configuration().harness.deployment.volume.mountpath
except Exception:
# not running inside a cloudharness deployment (e.g. unit tests)
Comment on lines
+74
to
+80
| set -e | ||
| name={{ .name | quote }} | ||
| ns={{ .root.Values.namespace | quote }} | ||
| marker=.cloudharness-volume-migrated | ||
| replicas=$(kubectl -n $ns get statefulset $name -o jsonpath='{.spec.replicas}') | ||
| for i in $(seq 0 $((replicas - 1))); do | ||
| pod=$name-$i |
| hostname = socket.gethostname() | ||
| filename = f"{time.strftime('%Y%m%d-%H%M%S')}-{hostname}.txt" | ||
| path = os.path.join(mountpath, filename) | ||
| content = (body or {}).get("content") or f"written by {hostname}" |
Comment on lines
+693
to
+697
| description: >- | ||
| HTTP methods (uppercase) the mapping applies to. Empty means all methods. | ||
| Passed through to the gatekeeper resource when `secured` is true. When | ||
| `deployment.statefulset` is true, uris declaring a write method | ||
| (POST/PUT/PATCH/DELETE) are also routed through the ingress to the leader |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (3)
deployment-configuration/helm/templates/ingress.yaml:16
- Ingress leader-routing can reference the "-rw" service even when auto Service generation is disabled, which would render an Ingress backend pointing to a Service that doesn't exist. Consider gating leader routing on
harness.service.auto(or otherwise guaranteeing the leader Service is created).
{{- /* White-listed uris bypass the gatekeeper; the ones declaring a write method are
routed to the leader (pod 0) service of a statefulset deployment. */}}
{{- $isWrite := and $app.harness.deployment.statefulset $mapping.methods (or (has "POST" $mapping.methods) (has "PUT" $mapping.methods) (has "PATCH" $mapping.methods) (has "DELETE" $mapping.methods)) }}
{{- $backendService := ternary (printf "%s-rw" $app.harness.service.name) $app.harness.service.name (not (not $isWrite)) }}
{{- $uri := $mapping.uri }}
deployment-configuration/helm/templates/ingress.yaml:45
- This block also routes to the "-rw" backend without checking whether the leader Service is rendered. If
harness.service.autois false, this will generate an Ingress backend to a missing Service.
{{- if and $app.harness.deployment.statefulset $app.harness.uri_role_mapping (not (and $app.harness.secured $secured_gatekeepers)) (eq $app.harness.gateway.pathType "Prefix") }}
applications/samples/backend/samples/controllers/test_controller.py:55
- Catching a broad
Exceptionhere can mask real runtime failures (e.g., config parsing issues) and silently fall back to/tmp/myvolume. It’s safer to catch the expected Cloud Harness configuration exception (and missing-volume cases) explicitly.
try:
from cloudharness.applications import get_current_configuration
mountpath = get_current_configuration().harness.deployment.volume.mountpath
except Exception:
# not running inside a cloudharness deployment (e.g. unit tests)
Comment on lines
+118
to
+120
| For a StatefulSet, an additional service named `<service-name>-rw` is automatically created that | ||
| always resolves to pod 0. This supports a single-writer pattern when running multiple replicas | ||
| over a shared volume: any pod can serve reads, but write requests are handled only by pod 0. |
| mountpath = "/tmp/myvolume" | ||
|
|
||
| hostname = socket.gethostname() | ||
| filename = f"{time.strftime('%Y%m%d-%H%M%S')}-{hostname}.txt" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes CH-282
Implemented solution
Added option in values.yaml
harness.deployment.statefulset=trueWhen statefulset is enabled:
How to test this PR
Set
harness.deployment.statefulset=truein one deployment and deploy itSanity checks:
Breaking changes (select one):
breaking-changeand the migration procedure is well described abovePossible deployment updates issues (select one):
alert:deploymentTest coverage (select one):
Documentation (select one):
Nice to have (if relevant):