Skip to content

deployment.md

Latest commit

 

History

History
461 lines (367 loc) · 21.9 KB

deployment.md

File metadata and controls

461 lines (367 loc) · 21.9 KB
Raw

Deployment

How a commit becomes a running service, and what each step is allowed to do on its own.

The rule the whole pipeline is built around: publishing an artefact is automatic, deciding what the world talks to is not. Building images on every green commit is harmless. Moving the enclave changes its measurement, and a measurement that changes without a decision somebody can be held to is exactly what the attestation exists to rule out.

The shape of it

flowchart TB
  push["push to main"]
  pr["pull request"]

  subgraph checks["checks.yml: all must pass"]
    frontend["frontend<br/>lint, typecheck, tests, build, smoke"]
    backend["backend<br/>lint, typecheck, tests, image build"]
    circuits["circuits<br/>witness tests, cached trusted setup"]
    contracts["contracts<br/>E2E with real Groth16 proofs"]
    tally["tally CLI"]
    circuits -- "artefacts" --> contracts
  end

  changed{"has anything changed?"}
  stop(["checked, never published"])
  site["publish the frontend<br/>(4EVERLAND)"]
  image["publish the backend image<br/>(GHCR, pinned by digest)"]
  auto{"AUTO_DEPLOY"}
  heroku["deploy to heroku"]
  phala["deploy to phala"]
  dns["DNS follows"]
  release["publish the release"]

  push --> checks
  pr --> checks --> stop
  checks --> changed
  changed -- "frontend/" --> site
  changed -- "what enters the image" --> image
  image --> auto
  auto -- "none" --> done(["stop"])
  auto -- "heroku" --> heroku --> dns
  auto -- "phala" --> phala --> dns
  site --> release
  image --> release
Loading

Manual entry points, at any time, for any published version:

  • deploy heroku: .github/workflows/deploy-heroku.yml
  • deploy phala: .github/workflows/deploy-phala.yml

Where things run

Host Hostname Carries the attestation Cloudflare proxy
Frontend 4EVERLAND / IPFS votain.app no off, DNS only
Backend Heroku container dyno or Phala Intel TDX enclave api.votain.app Phala only on
Chain local Hardhat through a named Cloudflare tunnel rpc.votain.app no tunnel

The frontend records must stay DNS only. With the proxy on, every push to main built and deployed on 4EVERLAND, and votain.app went on serving the previous deployment. Cloudflare's cache was not the cause: 4EVERLAND serves index.html with Cache-Control: no-cache and an ETag that is the IPFS root CID, and Cloudflare neither caches no-cache responses nor HTML by default. The cause is that 4EVERLAND only moves a custom domain onto each new deployment while it considers the domain verified, and it verifies by resolving the domain and expecting its own target. Proxied, the name resolves to Cloudflare, and the domain quietly stops following new builds. Their own troubleshooting advice is to disable the proxy.

Nothing is lost by it. 4EVERLAND already sits behind a CDN (responses carry Server: BunnyCDN), issues the certificate itself, and a static site has no origin to hide. One thing to watch: votain.app is the zone apex, and Cloudflare flattens an apex CNAME even when it is not proxied. If 4EVERLAND ever reports the domain unverified again, that flattening is the thing to suspect.

The backend is the opposite case, and works proxied: neither Heroku nor the enclave's ingress decides what to serve from what the name resolves to, so Cloudflare in front of them changes nothing about which version answers. See the dns.yml section below for why both of its CNAMEs are proxied.

api.votain.app is the switch. The frontend bakes that URL at build time and never learns which host answers it, so moving between Heroku and Phala is a DNS change and nothing else. Phala costs $42.34/month against $20 of credit, so it is stopped between sessions and Heroku carries the service meanwhile.

The workflows

checks.yml

Runs on every push to main and every pull request into it. Three test jobs, frontend, backend and contracts, each running the module exactly as it is run locally. A CI that runs different commands is a CI that goes green while the repository is broken for whoever clones it.

Everything that publishes anything is downstream of all three.

Change detection

The job has the backend changed decides whether there is anything to publish. The two halves use different baselines, and the difference is not an oversight:

Baseline Paths
Backend image the newest backend-v* tag only what the Dockerfile copies
Frontend the previous tip of main (github.event.before) all of frontend/

The backend leaves a mark: every published image creates a tag, so the question can be asked precisely. Has the backend changed since the last image? It survives a push of several commits, a re-run and a force push, all of which defeat a HEAD~1 comparison.

The frontend leaves no mark. 4EVERLAND builds from the branch and answers with a task id, not a version, so there is no frontend-v* to compare against. When the baseline is unusable, after a force push or on a newly created branch, it publishes anyway. Rebuilding a static site needlessly costs nothing; skipping a real change leaves votain.app stale, which is the failure nobody notices until somebody reports a bug that was fixed a week earlier.

Before this existed, the first three automatic versions were 0.1.1, 0.1.2 and 0.1.3, for commits that changed a workflow and a markdown file.

backend-image.yml

Builds the issuer image, publishes it to GHCR and attests it. Called by checks.yml, or run by hand, or fired by pushing a backend-v* tag.

The version grows by itself. The patch is bumped from the highest backend-v* tag, and the new tag is pushed back so every digest traces to a commit somebody can read. A version can also be named explicitly, for a minor or major bump that no rule can guess.

No version is ever reused. A digest pinned in the compose is only immutable if the tag beside it stays put, so the job refuses to overwrite a tag that exists. The flow this replaced republished 0.1.0 on every run.

There is no seed tag, deliberately. Pushing backend-v0.1.0 by hand would fire the tag trigger and republish 0.1.0, moving the registry tag away from the digest the enclave was attested on. Instead the sequence starts from the version the compose pins, which is by definition what is deployed.

deploy-heroku.yml

Takes a version, or defaults to the newest tag. Checks out that tag and rebuilds, because Heroku's registry rejects OCI manifests and the attested GHCR image is one. The Heroku image therefore has a different digest and carries no attestation, which is the honest shape of the two deployments rather than something to smooth over.

It distinguishes a release from a no-op. heroku container:release exits 0 either way; when the image is byte-identical to the running one it creates no release and says so only in a warning. The summary reports which of the two happened, and prints the release number before and after.

deploy-phala.yml

Takes a version, or defaults to the newest tag, and then:

  1. resolves the digest from the registry rather than trusting one typed by hand;
  2. verifies the provenance before the enclave sees it, because asking afterwards is asking too late;
  3. pins that digest in docker-compose.yml and commits it back, because the file an outsider verifies has to match what is running;
  4. hands the compose to Phala with the production environment;
  5. waits for the CVM to report running, then checks what it is actually running.

It asks the enclave, not the domain. An earlier version ended by curling https://api.votain.app/health and treating a 200 as success. That is worthless here: the domain points wherever DNS says, and with the CVM off it points at Heroku, so the check would have passed on a deployment that never reached the enclave. Phala reports docker_compose_hash, the sha256 of the compose it was given:

sha256sum docker-compose.yml

The same compose has two legitimate hashes, and this cost a failed run to learn. phala deploy --compose <file> sends the file as it is, trailing newline included. The dashboard takes the compose in a textarea, and a textarea eats the final newline. So a CVM first provisioned through the web UI reports the hash of the file without its last byte, and one deployed by the CLI reports it with. One byte, nothing about what runs, so the job accepts either.

sha256sum docker-compose.yml                          # deployed by the CLI
printf '%s' "$(cat docker-compose.yml)" | sha256sum   # via the dashboard

The job fails unless the CVM reports one of the two. It answers the only question worth asking. Does the enclave run what a reader of this repository can see? It says nothing about who happens to answer a hostname.

Switching hosts by hand

Two ways, and the difference is whether the code is also being updated.

# Move the hostname only. Both hosts are handled: the destination is brought
# up, the records change, and the one being left is stopped.
gh workflow run dns --ref main -f target=phala
gh workflow run dns --ref main -f target=heroku

# The same switch, plus rebuilding and releasing the backend first.
gh workflow run "deploy phala"  --ref main
gh workflow run "deploy heroku" --ref main

Both directions were run standalone and traced end to end. Switching to Phala started a stopped CVM, waited for it, wrote the three records, saw Heroku answer twice more from cache before Phala took over, and only then scaled the dyno to 0. Switching back scaled the dyno up from 0, removed the CAA and the TXT, saw Phala answer once more, and stopped the CVM.

dns is the quicker one and is enough when nothing about the backend changed. Add -f force=true to rewrite the records when they already name the target, which is how a record's shape is changed: proxied or not, a different TTL.

The deploy workflows take a version, and default to the newest backend-v* tag. They do not check whether anything changed: that filter belongs to the automatic path. A manual run deploys what you asked for, and if the image turns out identical Heroku says Nothing to release, but the host switch and the shutdown of the other still happen.

Before the defence, turn the automatic path off.

gh variable set AUTO_DEPLOY --body none --repo CodeInIA/votain

With AUTO_DEPLOY=heroku and the enclave serving, any commit to main that touches the backend will drag the hostname back to Heroku and stop the CVM, silently, with every test green, doing exactly what it was told. That is the right behaviour during development and the wrong behaviour in November.

Deploying the contracts to Amoy

Not part of any workflow: a contract deployment is a one-off decision, and so is the ceremony it rests on. In order, because each step needs the one before:

  1. Run the ceremony. cd circuits && PTAU=<public phase 1, at least 2^17> CEREMONY_ENTROPY=<secret> npm run build. The phase 1 comes from a public ceremony (Hermez or the PSE perpetual powers of tau; the historical Hermez URL answers 403, so take it from a mirror and check its hash). Discard the entropy afterwards: whoever keeps it can forge ballots. On Windows this needs the native circom compiler (see the README).
  2. Name the platform attester. ELIGIBILITY_ATTESTER_PRIVATE_KEY in backend/.env, or PLATFORM_ATTESTER_ADDRESS, and it must be the key the production backend signs with: it is frozen into every election.
  3. Deploy. PRIVATE_KEY and AMOY_RPC_URL in contracts/.env, then cd contracts && npm run deploy:amoy. The script refuses the development ceremony, a missing circuit build and a missing attester before its first transaction, so a refusal costs nothing. It writes contracts/deployments/amoy.json and mirrors it to frontend/src/lib/deployments/amoy.json: commit both, since the frontend is built from the repository and only local.json is ignored.
  4. Publish the proving files. frontend/public/circuits/ holds the wasm, zkey and vkey of THIS ceremony, tens of megabytes per zkey and too large for the repository, and CI cannot make them (it would have to see the ceremony's entropy). deploy:amoy ends by running npm run publish:circuits -- amoy in circuits/, which needs a public 4EVERLAND bucket and its key in circuits/.env. It refuses unless every verification key is the one the deployed verifier holds, uploads to a folder named after the ceremony, reads every file back cross-origin, and only then writes circuits.url into both manifests. The site reads it from there: no build variable to remember. Rerun it alone if the deploy got that far without the bucket configured; it skips what is already uploaded. CI's published circuits job repeats the download and the key check on every push and holds the frontend deploy if they fail. VITE_CIRCUITS_URL still overrides, for a preview build.
  5. Point the backend at it. Production reads addresses from its environment and ships no manifest: REGISTRY_ADDRESS and PAYMASTER_ADDRESS in PHALA_ENV (and on Heroku), with the relayer and registrar wallets funded with POL. The relayer is reimbursed by the paymaster; the registrar is not.

Secrets and variables

Repository secrets:

Secret Used by What it is
FOUREVERLAND_DEPLOY_HOOK checks.yml the URL that triggers a frontend build. The URL is the credential
HEROKU_API_KEY deploy-heroku.yml Heroku API key; also the registry password
PHALA_CLOUD_API_KEY deploy-phala.yml Phala Cloud API key
PHALA_ENV deploy-phala.yml the whole production env file, 25 variables
CLOUDFLARE_API_TOKEN dns.yml scoped to Zone:DNS:Edit on votain.app only

PHALA_ENV is not optional and the workflow refuses to start without it. The enclave's environment is sealed to the deployment; an update that supplies none can leave the backend without the keys it signs credentials with: ISSUER_PRIVATE_KEY, ELIGIBILITY_ATTESTER_PRIVATE_KEY, ENROLMENT_TAG_KEYS.

It is a copy. Rotating a key means updating the secret and the file in OneDrive. Two places, and nothing checks that they agree.

Repository variables:

Variable Values Effect
AUTO_DEPLOY unset / none / heroku / phala what follows a green commit that produced a new image

A variable rather than a secret, because which deployment is automatic is not confidential and anyone reading a run should be able to see it without permission to read secrets.

The 4EVERLAND deploy hook answers GET

Every other deploy hook, Vercel's, Netlify's and Render's, is a POST. 4EVERLAND's answers 503 {"code":500,"message":"SERVICE ERROR"} to a POST, which reads as "their service is down" or "your hook is dead" and is neither. The same URL succeeds as a GET and returns a queued task id.

The step checks the code in the body as well as the HTTP status, because this API reports failure inside a 200.

DNS follows the deployment

Both backends answer api.votain.app, so switching between them is a DNS change and nothing else. dns.yml makes that change part of the deployment instead of something to remember afterwards, and it is called by both deploy workflows.

It only acts on a real switch. It reads the current CNAME, works out which host it names, and if that is already the target it changes nothing. Rewriting the same records on every redeploy would be churn with a blast radius: each write is a chance to break resolution for a service that was working, and TLS issuance rides on two of these records.

The two sets are not symmetric.

Records Proxied
heroku one CNAME to the app's herokudns.com target yes
phala CNAME to the dstack gateway, CAA, and _dstack-app-address TXT yes

Switching to Heroku removes the CAA and the TXT. Left in place, the CAA authorises only the enclave's Let's Encrypt account over DNS-01, and Heroku's ACM could not issue a certificate for a name it now serves. Switching to Phala writes all three: the CAA authorises the ingress's ACME account, and the TXT tells the dstack gateway which app and port to route to. The CNAME alone resolves to a gateway that will not answer for this name.

Both CNAMEs are proxied. dstack-ingress wrote its own record unproxied, and that is how the enclave was first verified, so this is a deliberate departure. Nothing about the proxy should trouble it: DNS-01 validation runs on a TXT record, which is never proxied, and the origin already serves a certificate valid for this name, so Cloudflare reaches it over TLS like any other origin. If the ingress ever fails to renew a certificate, this is the first thing to suspect, and it is one boolean in dns.yml.

force: true rewrites the records even when they already name the right host. The guard compares hosts, so it cannot see that a record's shape should change, proxied or not or a different TTL, while the host stays the same. Without it, changing one boolean would mean switching away and back, which costs downtime and a certificate reissue.

The ingress does not restore these. Measured, by deleting the records and restarting the CVM and waiting five minutes for nothing: dstack-ingress writes its DNS records only on FIRST provisioning. That is why they are kept in this workflow rather than trusted to reappear, with a copy in OneDrive/UNI/4/TFG/.env/produccion/dns-api-votain-app.json.

The one we left is turned off, last. Paying for two backends to answer the same hostname is waste, and the one nobody is watching is the one that quietly runs up a bill. So a real switch ends by stopping the previous host: the Heroku dyno scaled to 0, or the CVM stopped.

Neither is deleted, and for Phala that distinction is the deployment. Deleting a CVM destroys the sealed environment with it: the keys the issuer signs credentials with are encrypted to that instance and do not come back. A stopped CVM keeps its disk, its identity and its measurement, and starts again in about a minute.

It happens only after the replacement has been seen to answer, not merely to resolve. A CNAME can be correct in the zone while the world still reaches the old host from cache. The check reads the reply and knows who sent it: Heroku's router stamps via: 1.1 heroku-router on every response and the enclave's ingress stamps nothing, so "the record says Phala" and "Phala answered" stay different claims. If the new host never answers, nothing is turned off and the run fails with the old one still serving.

The destination is brought up first, before the records move, so the hostname is never pointed at something switched off. dns.yml scales the dyno to 1 or starts the CVM and waits for it, and each deploy workflow does the same for its own host before handing over. The repetition is deliberate: both operations are idempotent, and it means dns.yml is safe to run on its own rather than only as the tail of a deployment.

Run standalone without it, a switch made after the other host had been turned off would move the name onto a dead service. The wait would then refuse to turn anything off. That is a safe failure, but one that leaves the hostname resolving to nothing.

The Heroku workflow checks /health in a separate job after the switch. Run before it, that check would be interrogating whichever host DNS happened to name, and a 200 from the enclave would say nothing about the release that just went out.

What a restart costs

Switching hosts restarts the backend, so it is worth being precise about what lives in memory. The backend has no database.

Sessions survive. The cookie carries the issued SD-JWT credential, and verifySession checks an Ed25519 signature, the expiry and revocation. There is no session table, so a voter who is signed in stays signed in across a restart and across a move between hosts.

That holds only while both hosts sign with the same ISSUER_PRIVATE_KEY. One host issues the credential and the other verifies it. Rotate that key in one place and not the other and every voter appears signed out the moment the hostname moves, with nothing in the logs to explain it. It is the real reason PHALA_ENV and the Heroku config have to be changed together.

Four things are held in memory, and they are not all the same:

What Losing it
organizerDomains.ts domain verification outcomes a cache; nothing
statusList.ts revocation slots, backed by the chain a cache; nothing
worldIdBridge.ts a sign-in in progress, 5 minute TTL that voter signs in again
eligibility/sessions.ts an attribute check in progress, 15 minute TTL that voter scans again

The last two are not caches, they are in-flight state, and they are in memory deliberately rather than for want of a database. Persisting an eligibility session would mean writing a record that links a World ID nullifier to an in-flight passport check, which is exactly the durable trace this project exists not to leave. Nothing durable is at stake either way: votes and enrolments are on the chain, and identity is in the signed credential.

Verifying it from outside

Nothing here asks to be trusted. Both links can be checked by anyone:

# The image came from this repository, built by this workflow, from this commit.
gh attestation verify \
  oci://ghcr.io/codeinia/votain-backend:<version> --repo CodeInIA/votain

# The enclave runs the compose in this repository.
# Either hash is valid: with the trailing newline if the CLI deployed it,
# without it if the CVM was provisioned through the dashboard.
sha256sum docker-compose.yml
printf '%s' "$(cat docker-compose.yml)" | sha256sum
phala cvms get --cvm-id <app-id> --json | jq -r '.docker_compose_hash'

The chain, with no link that asks for trust:

public commit → public build log → signed provenance → image digest
              → compose pinning that digest → TEE attestation of that compose

Heroku is outside this chain by construction, and the README says so. It is the deployment that stays up cheaply; Phala is the one that carries the argument.