Skip to content

fix: rework AWS private runner AMI build and module wiring - #24

Merged
hllvc merged 4 commits into
mainfrom
fix/aws-improvements
Aug 24, 2026
Merged

fix: rework AWS private runner AMI build and module wiring#24
hllvc merged 4 commits into
mainfrom
fix/aws-improvements

Conversation

@hllvc

@hllvc hllvc commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes three defects in the AWS Private Runner modules — an AMI that rebuilt on every apply, a console URL derived incorrectly outside EU1/US1, and a single_runner module that could not be composed with runner_group — and adds a quickstart example that wires all three modules into a single apply.

Motivation & Context

Deploying a private runner meant three separate applies with outputs copied by hand between them, and the result was not stable: null_resource.packer_build triggered on timestamp(), so every apply rebuilt the AMI from scratch and, because the AMI ID was read straight out of the build log, replaced the runner instance along with it. A no-op apply cost a full Packer build and a runner restart.

Two smaller problems surfaced while building the combined example:

  • The web console URL was produced by stripping api. out of the API host. That happens to work for api.app.stackguardian.io and api.us.stackguardian.io, but not for QA, where the API is at testapi.qa.stackguardian.io and the console at dash.qa.stackguardian.io. The wrong host landed in runner_group_url and, more consequentially, in the S3 bucket's CORS allowed_origins, so console-driven uploads were rejected.
  • single_runner looked its runner group up through the stackguardian provider data sources, so it could only target a group that already existed and re-read the token on every plan. It could not be composed with runner_group in one configuration.

Changes Made

Packer AMI lifecycle (aws/packer/)

  • Replaced the timestamp() trigger on null_resource.packer_build with packer_config.rebuild_ami_token, so the AMI is built once per state and reused thereafter. The token is a free-form string rather than a boolean, so unsetting it does not trigger another build.
  • Added terraform_data.ami_id to record the built AMI ID in state. Plans no longer depend on packer_manifest.log being present, which previously broke on a fresh checkout or a CI runner.
  • data.external.packer_ami_id now returns an empty ID instead of failing the plan when the log is missing, and resolves the log through path.module.
  • build_ami.sh checks for the artifact,0,id line before succeeding. tee masks Packer's exit status, so a failed build previously recorded as successful and stuck until the token changed.
  • null_resource.ami_cleanup keys off the recorded ID, so a destroy never deregisters an AMI this state did not build.
  • Added sg_runner.pre_release to bake the newest sg-runner pre-release into the AMI, falling back to the latest stable release when none is published.
  • Raised required_version to >= 1.4.0 for terraform_data.

Runner group (runner_group/)

  • Replaced the replace(api_uri, "api.", "") derivation with an explicit API-host-to-console-host map, used by both runner_group_url and the S3 CORS allowed_origins.

Single runner (aws/single_runner/)

  • Dropped the stackguardian_runner_group, stackguardian_runner_group_token, and aws_s3_bucket data sources; the module now takes runner_group_token as a sensitive input.
  • Added runner_group_token to input_schema.json (properties and required) and ui_schema.json as a password input documenting the ${secret::NAME} reference form. Without this the no-code form offered no field for a now-required variable.

Quickstart example (examples/aws/quickstart/)

  • New root module combining runner_group, packer, and single_runner. Four required values, one apply, and the runner registers itself.
  • README covering the AMI-reuse behaviour, the VPC interface-endpoint rule that otherwise makes jobs hang on plan with no error, and the example's deliberate limits (public subnet only, single runner, local state).

Repo hygiene

  • Ignore tfplan and .claude/settings.local.json. The existing pattern was *.tfplan, which does not match a plan file written as plain tfplan, so a binary plan could be committed by accident.

Testing

  • tofu fmt -check clean across the example and changed modules
  • Both single_runner schemas parse, and required now matches exactly the variables that have no default
  • Example module source paths resolve; tfvars template parses
  • tofu init && tofu apply in examples/aws/quickstart/ against a real VPC
  • Second apply shows no diff and does not rebuild the AMI or replace the instance
  • Bumping rebuild_ami_token builds exactly one new AMI and deregisters the superseded one
  • Destroy leaves AMIs from other deployments untouched
  • Runner registers and appears active in the runner group
  • On QA (testapi.qa.stackguardian.io), runner_group_url and the S3 CORS origin both resolve to dash.qa.stackguardian.io
  • No-code form renders the new Runner Group Token field and accepts a ${secret::NAME} reference

Risks & Edge Cases

  • Breaking change for single_runner consumers. runner_group_token is required and has no default. Existing configurations and no-code deployments must supply it; previously the module discovered it on its own.
  • api_uri is now constrained. The map lookup accepts only the three known platform hosts. Any other value fails the plan rather than silently producing a wrong console URL and CORS origin. This is intentional, but it is stricter than before.
  • One rebuild on upgrade. On the first apply after this change, the packer_build trigger switches from timestamp() to the token, which replaces the resource once. Expect a single AMI rebuild and one runner instance replacement, then stability.
  • Minimum version raised to Terraform 1.4 / OpenTofu 1.6 for terraform_data.
  • cleanup_amis_on_destroy now only ever touches AMIs this state built. Images previously created by repeated applies of older revisions are not adopted and will need manual cleanup — cleanup_commands output has the CLI for it.

Deployment Notes

  • Existing single_runner deployments must add runner_group_token before the next apply.
  • Plan the first apply after this change during a window where a single AMI rebuild and runner replacement is acceptable.
  • No new secrets or IAM permissions are required beyond what the modules already needed.

hllvc added 4 commits August 24, 2026 11:25
Packer rebuilt the AMI on every apply, because null_resource.packer_build
triggered on timestamp(). That cost a full build each run and, since the AMI
ID came straight from the build log, replaced the runner instance every time
too.

Build once per state instead. The ID is recorded in terraform_data.ami_id and
read from there on later plans, so a fresh checkout or a CI runner no longer
depends on packer_manifest.log being present. Set packer_config.rebuild_ami_token
to any new value to build a fresh AMI; leaving it alone never rebuilds. It is a
free-form string rather than a flag so unsetting it does not trigger another
build.

Also:
- build_ami.sh now checks for the artifact line, since tee masks packer's exit
  status and a silent failure would otherwise be recorded as a successful build
- ami_cleanup keys off the recorded ID, so destroy never deregisters an AMI this
  state did not build
- new sg_runner.pre_release bakes the newest sg-runner pre-release into the AMI,
  falling back to latest stable when none is published
- requires terraform >= 1.4 (opentofu >= 1.6) for terraform_data
The console URL was derived by stripping "api." out of the API host. That works
for app.stackguardian.io and us.stackguardian.io but not for QA, where the API
lives at testapi.qa.stackguardian.io and the console at dash.qa.stackguardian.io.
The wrong host ended up in runner_group_url and, more importantly, in the S3
bucket's CORS allowed_origins, so browser uploads from the console were rejected.

Map API host to console host explicitly instead. The console host is not
derivable from the API host in every region, so an unlisted api_uri now fails
the plan rather than silently producing a bad origin.
The module looked the runner group up itself through the stackguardian provider
data sources, which meant it could only be used against a runner group that
already existed, and it re-read the token on every plan.

Take runner_group_name and runner_group_token as inputs instead and drop the
data sources. The module can now be composed directly with runner_group in one
apply, with the token passed in memory as a sensitive value.

runner_group_token is required, so add it to the template schemas as well -
without it the no-code form offers no field for the token and the apply fails
on a missing variable. It renders as a password input and documents the
${secret::NAME} reference form, since the value should not be typed literally.
Deploying a runner meant three separate applies with outputs copied by hand
between them. examples/aws/quickstart wires runner_group, packer, and
single_runner into one root module: fill in the API key, org, VPC, and subnet,
apply once, and the runner registers itself.

The README covers the AMI-reuse behaviour, the VPC interface endpoint rule that
otherwise makes jobs hang on plan with no error, and where the example stops -
public subnet only, single runner, local state.

Also ignore tfplan and .claude/settings.local.json. The existing pattern was
*.tfplan, which does not match a plan file written as plain "tfplan", so a
binary plan could be committed by accident.
@hllvc hllvc self-assigned this Aug 24, 2026
@hllvc
hllvc requested a review from arunim2405 August 24, 2026 10:04
@hllvc
hllvc marked this pull request as ready for review August 24, 2026 10:04
@hllvc
hllvc merged commit d45046e into main Aug 24, 2026
1 check passed
@hllvc
hllvc deleted the fix/aws-improvements branch August 24, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants