Add AWS Vault AMI, user-data, and launch template

[amitslog]

This slice bakes Vault CE and vault-utils into an AMI and boots a node without manual steps. User-data writes one env file. A baked configuration service runs after cloud-init, writes the AWS-specific Vault config, and exits. Systemd ordering then starts Vault, runs bootstrap once, and starts health on 8210. Bootstrap failure keeps health and snapshots down. Scheduled snapshots run only on the Raft leader. A launch template attaches the AMI, instance profile, node security group, and user-data. ASG and NLB are not in this PR.

PR Summary

• Packer build lives in vault-node/. vault-node/files/ is cloud-neutral (base vault.hcl, systemd units). vault-node/aws/vault-node-configure is the only AWS-specific file.
• AMI is x86_64 AL2023 (nullstone-vault-*), override with ami. Default instance type is t3.micro.
• User-data writes /etc/vault.d/vault-utils.env and nothing else.
• vault-configure.service runs after cloud-final.service, writes /etc/vault.d/cloud.hcl and /etc/vault.d/node.env, and exits. All units are enabled during the bake.
• Launch template uses IMDSv2 and the vault-cluster join tag.
• snapshot schedule skips non-leaders.

Depends on

• PR Add aws-ec2-vault-cluster connections, IAM, and security groups #3 (connections, IAM, Secrets Manager, security groups), already on develop.
• PR Add AWS vault-utils: Secrets Manager, KMS bootstrap, and Raft health #4 (bootstrap aws, health on 8210, S3 snapshots), already on develop.
• A baked nullstone-vault-* AMI in the account, or ami set. tofu plan fails without one of those.
• A Nullstone workspace with network, snapshots_bucket, and unseal_key.

How I validated

go vet, gofmt, and go test -short ./... pass. tofu fmt -check and tofu validate pass. shellcheck and bash -n pass on provision.sh and vault-node-configure. systemd-analyze verify passes on all five units on AL2023. vault-node-configure was run end to end in an AL2023 container with a stubbed metadata service, and the generated cloud.hcl was loaded by the pinned Vault 2.0 image alongside the baked vault.hcl. Packer bake and a live EC2 boot were not run.

Review Note 1

Unit tests cover the leader check and existing vault-utils paths. This branch does not change Compose. Live KMS unseal, Secrets Manager bootstrap, and NLB health on a running instance are not covered.

Review Note 2

Review feedback applied in e584337 and e170f3b. Base vault.hcl and systemd units moved into Packer. User-data reduced to writing the env file. Ports 8200 and 8201 hardcoded. AMI is x86_64 with default t3.micro.

Review Note 3

Second review round applied in 4d390d4 and c50634d. Packer build moved to vault-node/. Path unit removed; vault-configure.service is ordered after cloud-final.service and no longer calls systemctl. disable_mlock = false added. Operator policy grants read on sys/storage/raft/autopilot/state. Node security group allows 8200 node to node. kms:GenerateDataKey added for SSE-KMS snapshots. Configure script reads env values individually instead of sourcing. Nil index in the platform fallback fixed. All five units enabled at bake, with vault-snapshot.service on Restart=on-failure so an empty schedule leaves it stopped.

Deferred with the reviewer's own scoping: multi-node init race to the ASG slice, api_addr on the NLB DNS name to the NLB slice.

Next

Internal NLB with the target health check on 8210, then the ASG (surge min_healthy_percentage = 100, max_healthy_percentage = 134). No lifecycle hooks on day one.

[BSick7]

I left several comments.

In general, we should adjust this to eliminate user-data as much as possible.
The node should automatically start and connect to the vault cluster during bootup.

We should also reuse as much as we can across cloud platform VMs.

[amitslog]

User-data now only writes the env file. A baked path unit runs configure, writes /etc/vault.d/cloud.hcl, and starts Vault so the node joins on boot. Units and base vault.hcl are cloud-neutral. AWS seal and join stay in the AWS configure script.

[BSick7]

Let's move the packer build up to /vault-node/ dir instead of /aws/aws-ec2-vault-cluster/packer/.

I don't think this is going to work in its current form.
vault-node-configure appears to run when building the packer image, but it pins down instance-specific info into the vault config (e.g. private ip, instance id, region, etc.).
I would imagine the first systemd service to run on bootup would be a configuration service that writes this vault hcl file.

Here are some other issues I found:

1. Vault will not start on the node. The rendered config in templates/user-data.sh.tpl has no disable_mlock setting. Since Vault 1.20 that setting is mandatory with Integrated Storage, and the pinned 2.0 image refuses to boot without it. I verified this by running the same image with the equivalent config:
   ERROR: disable_mlock must be configured 'true' or 'false': Mlock prevents
   memory from being swapped to disk ... Prior to Vault 1.20, disable_mlock
   defaulted to 'false' when the value was not set explicitly.
   Result on a real instance: the health poll in user-data times out after two minutes, cloud-init exits 1, and nothing else runs. Fix is one line in the HCL heredoc. Given the unit already grants CAP_IPC_LOCK and LimitMEMLOCK=infinity, disable_mlock = false is the intended production value.

2. Health checks and leader-gated snapshots are denied by the operator policy. Both health serve and the new snapshot gate read sys/storage/raft/autopilot/state with the operator token, but policies/templates/operator.hcl.tpl only grants raft/snapshot and raft/configuration. The health endpoint will return 503 forever, and since user-data sets VAULT_RAFT_NODE_ID for the snapshot service too, every scheduled snapshot will log "snapshot skipped" and never run. The health half predates this commit, but the snapshot regression is new. Add a read grant for the autopilot state path.

3. Nodes cannot reach each other on 8200, so retry_join cannot work. Raft join first calls the leader's API port, then moves to 8201. The nodes security group only allows 8200 from the NLB and 8201 node-to-node. Harmless while cluster size is 1, but it will block the multi-node cluster the ASG work is heading toward.

4. Multi-node init race is only partly handled. The new fallback in bootstrap.go covers the narrow window where a node joins between the seal-status check and the init call. If two fresh nodes boot before either is initialized, both will init and form separate clusters. The second node also races on token issuance and root revocation. Bootstrap should be leader-only or serialized before the ASG lands.

5. Snapshots to a KMS-encrypted bucket will fail. The SnapshotKey statement in iam.tf grants Encrypt, Decrypt, and DescribeKey, but S3 SSE-KMS PutObject needs kms:GenerateDataKey. Pre-existing, but this commit is the first to run snapshots unattended.

6. First boot deadlocks on the path unit. vault-configure.service declares Before=vault.service, and vault.service declares After=vault-configure.service. The configure script then runs a synchronous systemctl enable --now vault from inside ExecStart.

7. Any non-empty backup schedule breaks the configure script. The env file is written in systemd EnvironmentFile format but is now also sourced by bash with set -a. A standard cron value like 0 3 * * * is parsed by bash as the command 3 with an environment prefix, and set -e aborts the script with exit 127. I reproduced this locally. Since the same file also feeds systemd, quoting the value in user-data would fix bash but systemd then keeps the quotes stripped, so quoting is safe there. Alternatively, read the four values the script needs with grep rather than sourcing.

8. Nil index panic in the new platform fallback. In cmd/vault-utils/main.go the azure, gcp, and default branches still format args[0] after the switch was changed to use platform. With VAULT_PLATFORM=gcp and no positional argument, that indexes an empty slice and panics instead of returning the "not implemented" error.

9. Path unit can fire on a partially written env file. The path unit triggers on existence, and user-data creates the file with a heredoc then chowns and chmods it. The configure script can start before the write and permission steps finish. The window is small, but the failure mode is the same silent hang as item 1. Writing to a temp name and renaming into place closes it.

[amitslog]

Revision 3:

Design. vault-node-configure did not run during the Packer build, the build only installed it and ran bash -n on it. The path unit made the boot trigger implicit though, so the concern was fair, and it is gone. vault-configure.service now runs after cloud-final.service, writes /etc/vault.d/cloud.hcl and /etc/vault.d/node.env, and exits. Systemd ordering starts everything after that. That is the configuration service you described.

Packer location. Moved to vault-node/. vault-node/files/ holds the cloud-neutral image content, the base vault.hcl and the five systemd units. vault-node/aws/vault-node-configure is the only AWS-specific file, so another platform adds a sibling directory instead of forking the units.

1. disable_mlock = false is in the baked vault.hcl. I reproduced your error against the pinned 2.0 image and confirmed the fix boots with Mlock: supported: true, enabled: true under CAP_IPC_LOCK.
2. Added a read grant on sys/storage/raft/autopilot/state to the operator policy. Verified against a live Vault: the old policy returns deny on that path and the new one returns read, while snapshot-force, tenant KV data, and the operator policy itself still return deny.
3. Added ingress and egress on 8200 scoped to the nodes security group. Egress was needed as well, since node egress only covered 443 and 8201.
4. Multi-node init race is not in this PR. There is no ASG yet, so nothing launches a second node. It lands in the ASG slice.
5. Added kms:GenerateDataKey to the conditional snapshot key statement, still scoped to that key ARN.
6. The configure script no longer calls systemctl at all. Ordering is cloud-final, then vault-configure, then vault, then vault-bootstrap, then vault-health and vault-snapshot. systemd-analyze verify passes on all five units on AL2023.
7. The script no longer sources the env file. It reads the four values it needs individually, so a cron expression cannot be executed. The value is also quoted in user-data, which systemd strips. I reproduced the exit 127 and confirmed the fix.
8. Both branches use platform now. Confirmed the old binary panicked and the new one returns the not-implemented error.
9. The path unit is gone, so nothing watches the file. Ordering after cloud-final.service means user-data has completed before configure starts. User-data also sets mode and owner before writing content. I went with ordering rather than temp-and-rename because it removes the race by construction.

Also ran vault-node-configure end to end in an AL2023 container with a stubbed metadata service, and loaded the generated cloud.hcl with the baked vault.hcl into the pinned Vault image to confirm it parses.

Packer bake and a live EC2 boot are still not covered.

[amitslog]

One more in 1534eee, found while validating the bake path locally now that Packer is installed here. The shell provisioner had no execute_command, so provision.sh would have run as ec2-user and failed on the first useradd. It now runs through sudo -E sh -c. I confirmed both sides in an amd64 AL2023 container: the old form fails with useradd: Permission denied, the new form runs provision.sh to completion, installs Vault, and lays down all five units. Default vault_version is now 2.0.4 to match the pinned image used for the mlock and config checks.

[BSick7]

There are several security group rules like this.
Let's switch to self = true.

[BSick7]

Since we're tagging it, let's use the tag instead of a name wildcard:

filter {
  name   = "tag:Name"
  values = ["nullstone-vault"]
}

[BSick7]

Looking really good, left a few smaller notes.

Can we also add a github action workflow to build these AMIs against multiple regions?
Let me know what env vars to add to configure publishing to the nullstone org.