Skip to content

Collect every release artifact under one convention - #433

Closed
holodorum wants to merge 5 commits into
kson-org:mainfrom
holodorum:ci/release-artifact-collection
Closed

Collect every release artifact under one convention#433
holodorum wants to merge 5 commits into
kson-org:mainfrom
holodorum:ci/release-artifact-collection

Conversation

@holodorum

@holodorum holodorum commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

What this does

store-native-artifacts stored kson-lib/build/bin/nativeKson/releaseShared and releaseStatic.
Nothing writes those paths — kson-lib declares only jvm() and js() targets — so all three
build-* jobs have been storing nothing, while the native library that does exist (in
kson-lib/build/kotlin/compileGraalVmNativeImage) and the CLI binary that check already builds on
all three platforms were both thrown away with the build directory.

A new ./gradlew packageReleaseArtifacts stages this platform's share of a release into
build/release-artifacts:

file contents
kson-lib-shared-<arch>-<os>.tar.gz the native-image output directory, verbatim
kson-cli-<arch>-<os>.tar.gz the CLI binary
SHA256SUMS checksums for both

The <arch>-<os> tokens are not ours to pick. lib-rust/kson-sys/build.rs downloads
kson-lib-shared-{arch}-{os}.tar.gz from the kson-lib-X.Y.Z release on kson-org/kson-binaries
and unpacks it straight into its OUT_DIR, so the kson-lib archive carries that directory's
contents at the archive root — unpacking it leaves exactly what build_kson_from_source copies
there. The tokens live in BinaryArtifactPaths beside a note naming whose contract they are, with
tests pinning the spellings against build.rs. The CLI archive follows the same convention so there
is one to learn rather than two.

CI packages inside gradle-core-tasks, before verifyCleanCheckout, so every build proves
packaging leaves the checkout clean; all six artifact-producing jobs now store through one
store-release-artifacts command into release-artifacts/<platform>. Branch filters and caching are
untouched.

Also fixes :tooling:cli:buildNativeImage declaring kson as its output on Windows, where
native-image writes kson.exe: packaging needs the real name, and the task could never be up to
date there.

What this means for #423

#423's Step 1 is a workaround for exactly the gap this closes — it has the release manager download
three Python wheels and unzip jni_simplified.h plus the platform library out of each. Those
archives now come out of CI ready-made, so this PR rewrites that step and corrects the facts around
it:

  • Step 1 becomes three downloads. The wheel-repackaging for loop is gone. The
    KSON_LIB_VERSION check (and its "every downstream cargo build 404s" warning), the local
    KSON_PREBUILT_BIN_DIR sanity build, and the gh release create invocation all stay.
  • The job/artifact table is corrected. The libraries come from build-linux-amd64 /
    build-macos-arm64 / build-windows-amd64 into release-artifacts/<platform>, not from
    test-python-sdist-* into python-*. Every name was checked against .circleci/config.kson on
    this branch.
  • "Exactly two files" is corrected. A wheel-extracted archive held jni_simplified.h plus the
    library. The CI archive is the native-image output directory verbatim — on macOS that is
    graal_isolate.h, graal_isolate_dynamic.h, jni_simplified.h and libkson.dylib.
  • The workflow prerequisite is corrected from a green build-python-and-test to a green
    build-all, which is the workflow whose jobs build these libraries.

#423's publish ordering, prerequisites, cargo publish --dry-run steps, the "published crates can
only be yanked, never changed" warning and the crates.io verification are kept as they are.

Deliberate gaps

  • No GitHub Release automation. Uploading to kson-org/kson-binaries stays a documented manual
    gh release create.
  • No CLI distribution channel. The binaries are collected; where they get published is still a
    todo in the doc, because nothing has been chosen.
  • No signing or notarization of the CLI binary or the shared library.
  • No new platform jobs. build.rs maps tokens for arm64/amd64 across
    linux/macos/windows, but CI produces only amd64-linux, arm64-macos and amd64-windows.
    The other three — arm64-linux, amd64-macos and arm64-windows — 404 on download, and
    consumers there must set KSON_ROOT_SOURCE_DIR or KSON_PREBUILT_BIN_DIR themselves (the doc
    says so). Adding jobs for them is a separate change.

Verified locally (macOS arm64)

  • cd buildSrc && ./gradlew check — green, including 9 new tests (Sha256SumsTaskTest,
    BinaryArtifactPathsTest)
  • ./gradlew check — green on this base (146 tasks), so replaying onto rust-release broke
    nothing
  • ./gradlew transpileCircleCiConfigTask — regenerates config.yml with no diff, so it is
    committed in sync with config.kson
  • ./gradlew packageReleaseArtifacts — produces kson-lib-shared-arm64-macos.tar.gz (four files,
    library mode 0755), kson-cli-arm64-macos.tar.gz (kson, mode 0755) and SHA256SUMS;
    shasum -a 256 -c SHA256SUMS passes
  • ./gradlew verifyCleanCheckout — clean, both before and after packaging
  • The doc's own sanity check, run for real: unpacking kson-lib-shared-arm64-macos.tar.gz and
    building with KSON_PREBUILT_BIN_DIR pointed at it compiles kson-sys and kson-rs, so the
    archive really is a drop-in for what build.rs expects

Linux and Windows packaging is unverified locally — that is what the build-* jobs on this branch
are for.

joeslice and others added 5 commits August 25, 2026 11:30
Included is an untested draft version of release steps that might be useful when shipping a new release of the rust artifacts to crates.io. This should be validated and corrected with Adolfo's help before carrying on.
Simplifying and improving the instructions while testing them.
This step was missed 17d124d and 3c147dd.

`cargo publish` was the first to regenerate and notice an error here.
The prior example was no longer accurate since requiring `options` instead of a boolean. This new example demonstrates some indentation options and also a bit more about the json->kson pipeline.
`store-native-artifacts` stored `kson-lib/build/bin/nativeKson/releaseShared`
and `releaseStatic`. Nothing writes those paths: kson-lib declares only `jvm()`
and `js()` targets, so there is no Kotlin/Native build to produce them, and all
three `build-*` jobs have been storing nothing. Meanwhile the native library
that does exist -- in `kson-lib/build/kotlin/compileGraalVmNativeImage` -- and
the CLI binary that `check` already builds on all three platforms were both
thrown away with the build directory.

Add `./gradlew packageReleaseArtifacts`, which stages this platform's share of
a release into `build/release-artifacts`:

  kson-lib-shared-<arch>-<os>.tar.gz  the native-image output directory, verbatim
  kson-cli-<arch>-<os>.tar.gz         the CLI binary
  SHA256SUMS                          checksums for both

The `<arch>-<os>` tokens are not ours to pick. `lib-rust/kson-sys/build.rs`
downloads `kson-lib-shared-{arch}-{os}.tar.gz` from the `kson-lib-X.Y.Z` release
on kson-org/kson-binaries and unpacks it straight into its `OUT_DIR`, so the
kson-lib archive carries that directory's contents at the archive root --
unpacking it leaves exactly what `build_kson_from_source` copies there -- and
the tokens live in `BinaryArtifactPaths` beside a note naming whose contract
they are. The CLI archive follows the same convention so there is one to learn
rather than two.

CI packages inside `gradle-core-tasks`, before `verifyCleanCheckout` so every
build proves packaging leaves the checkout clean, and all six jobs now store
through one `store-release-artifacts` command into `release-artifacts/<platform>`.
Branch filters and caching are untouched.

Also fixes `:tooling:cli:buildNativeImage` declaring `kson` as its output on
Windows, where native-image writes `kson.exe`: packaging needs the real name,
and the task could never be up to date there.

`docs/release_process.md` gains a table of which job produces what and where to
download it, and fills in the collection half of the lib-rust and tooling/cli
sections. Publishing the CLI stays a todo: no distribution channel has been
chosen for it.

The lib-rust process this builds on (kson-org#423) opened with a workaround for exactly
the gap closed here: download three Python wheels and unzip `jni_simplified.h`
and the platform library out of each. Its Step 1 becomes three downloads of
archives CI now builds ready-made, and the facts around it are corrected with
it -- the libraries come from the `build-*` jobs rather than
`test-python-sdist-*`, they land in `release-artifacts/<platform>`, and an
archive holds the native-image output directory verbatim (four files on macOS)
rather than the two a wheel could supply. That document's publish ordering,
`KSON_LIB_VERSION` check, crates.io steps and their warnings are kept as they
are.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xh2RGwbRjLofpWRJK9cT5z
@holodorum holodorum closed this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants