ai: prepare for agentic workflows

.github/scripts/external-packages-license-check.go

@@ -59,7 +59,7 @@ func run() error {
 	}
 
 	// Check the internal dependencies of the external packages.
-	output, err := exec.Command("sh", "-c", "go list -deps -test ./public/*").Output()
+	output, err := exec.Command("sh", "-c", "go list -deps -test ./public/...").Output()
 	if err != nil {
 		return err
 	}

.kb/agents.md

@@ -0,0 +1,44 @@
+# Preface
+
+This repository follows strict conventions for organizing agent-oriented knowledge documents, and this document is required reading for any agent that wants to read or write these.
+
+Read the top-level `.kb/agents.md` before continuing below.
+
+
+# Overview
+
+Every directory in this repository, including the root, may have its own `AGENTS.md` file and `.kb/` subdirectory. The `AGENTS.md` file provides a general view for the directory, while more specific information is found in `.kb/*.md` files with dashed lowercase names (e.g. `.kb/special-relativity.md`).
+
+The design of this structure has the following key goals:
+
+- **Mechanical** - Agents are the main actors reading and writing the knowledge base.
+- **Generic** - Benefits any agentic workflow, no matter the editor or platform.
+- **Distilled** - Avoids the use of verbose task or plan logs that pollute the context window.
+- **Hierarchical** - Avoids excessive information in a single place that also pollutes the context window.
+- **Human** - Information is readily available and reviewable in a friendly format.
+
+
+# Important
+
+- Read local `AGENTS.md` files upon navigating directories.
+- Keep the `AGENTS.md` and `.kb/*.md` files updated whenever there is something relevant to be documented or updated. The top-level `.kb/agents.md` must NOT be edited unless explicitly requested.
+- Follow the header conventions outlined below. Only the _Preface_ header is required, and the other headers should be omitted if empty or trivial.
+
+
+# Headers
+
+The following are the ONLY top-level headers allowed across the `.kb/*.md` files in this repository, to maintain semantic standardization across projects. Keep the ordering of created sections the exact same you see below.
+Sub-headers are okay.
+
+- _Preface_ - A brief introduction outlining the scope and relevance of a specific `*.md` file. This section MUST be at the top of every `.kb/*.md` file so agents can easily grep for it, and the last line of this section MUST be "Read the top-level `.kb/agents.md` file before continuing below." so rules are followed.
+- _Overview_ - High-level summary of the directory, subsystem or knowledge base layout at large. Do NOT use this to list files or directories.
+- _Important_ - Essential directives for the agent outlining critical constraints, behaviors, or rules.
+- _Headers_ - Global registry of header definitions, uniquely hosted at the root `.kb/agents.md` (this file). Do NOT use this header in any other document.
+- _Architecture_ - Structural design details or boundary explanations for a given component. Only use this for software architecture concepts, NOT for directory outlining. Also avoid using this as a code reference (keep that in the code itself).
+- _Directory_ - Only in `AGENTS.md` files, it briefly outlines the contents and structure of the directory the `AGENTS.md` file is in, and potentially nested small directories that do not justify their own `AGENTS.md` file. For `.kb/*.md` files, use _Documents_ instead.
+- _Documents_ - Only in `AGENTS.md` files, it outlines the content of `.kb/*.md` files and also immediately nested `AGENTS.md` child files, to aid agent navigation. It MUST be the last section in the file. It's okay to also mention such filenames inline when they are relevant elsewhere in the text.
+
+For the _Directory_ and _Documents_ listings, format items as a dashed list starting with the file or directory name surrounded by backticks, a dash, and then a brief description:
+```
+- `filename` - Terse summary.
+```

AGENTS.md

@@ -0,0 +1,45 @@
+# Preface
+
+This repository hosts Chisel, a software tool for carving and cutting Debian packages. This `AGENTS.md` is the root index into the dispersed knowledge base; read it to orient yourself, then follow the links below for details.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+
+# Overview
+
+Chisel cuts fine-grained slices out of Debian packages. It takes a selection of slices defined in chisel-releases, resolves their dependencies, downloads the corresponding packages from Ubuntu archives, extracts the selected files into a target root filesystem, runs Starlark mutation scripts, and generates manifests at paths requested by the selected slice definitions. The codebase is split across command entry points, an internals tree holding the carving pipeline, and public packages defining the manifest format.
+
+
+# Important
+
+- Prefer deterministic, predictable behavior that fails on the side of correctness and preserves existing user content by default.
+- Keep Chisel as independent as practical from incidental package archive contents.
+- Treat changes to the chisel-releases format and the public manifest packages as compatibility-sensitive.
+
+
+# Directory
+
+- `cmd/` - Binary entry points for the `chisel` application.
+- `internal/` - Core internal packages: slice orchestration, setup, extraction, archive, cache, and supporting utilities.
+- `public/` - Public API packages for external consumers.
+- `snap/` - Snap packaging configuration for Chisel.
+- `tests/` - End-to-end integration test suite.
+- `docs/` - Documentation assets.
+- `go.mod` - Go module definition.
+- `go.sum` - Go dependency checksums.
+- `spread.yaml` - Spread test configuration.
+- `workshop.yaml` - Workshop configuration.
+- `.golangci.yaml` - golangci-lint configuration.
+- `README.md` - Project readme.
+- `SECURITY.md` - Security policy.
+- `LICENSE` - Project license.
+
+
+# Documents
+
+- `.kb/agents.md` - General rules for the knowledge base reading and writing.
+- `cmd/AGENTS.md` - CLI entry points.
+- `internal/AGENTS.md` - Core internal packages.
+- `public/AGENTS.md` - Public API packages.
+- `snap/AGENTS.md` - Snap packaging.
+- `tests/AGENTS.md` - Integration testing.

cmd/AGENTS.md

@@ -0,0 +1,25 @@
+# Preface
+
+This document serves as the local knowledge base index for the `cmd/` directory. It outlines the scope and purpose of the entry points for the `chisel` binary.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Overview
+
+The `cmd/` directory contains the main binary entry points for the `chisel` application. Instead of housing core business logic, this directory is responsible for initializing the application, wiring up the CLI subsystem, and handling version information. Core logic is deferred to the packages in `internal/`.
+
+# Directory
+
+- `chisel/` - Contains the primary executable entry point for the `chisel` application.
+- `chisel/main.go` - Application bootstrap and CLI initialization.
+- `chisel/cmd_cut.go` - Implements the `cut` command, the primary operation that extracts slices into a target root filesystem.
+- `chisel/cmd_find.go` - Implements the `find` command for searching available slices by name or package.
+- `chisel/cmd_info.go` - Implements the `info` command for displaying detailed metadata about specific slices.
+- `chisel/cmd_debug.go` - Implements the `debug` subcommand group for additional, development-oriented commands without stability guarantees.
+- `chisel/cmd_debug_check_release_archives.go` - Implements `debug check-release-archives`, which validates the release with packages content from the archive.
+- `chisel/cmd_version.go` - Implements the `version` command.
+- `chisel/cmd_help.go` - Implements the `help` command.
+- `chisel/helpers.go` - Shared CLI utilities used across commands.
+- `chisel/log.go` - Logging setup for the CLI layer.
+- `mkversion.sh` - Shell script used for generating version information during builds.
+- `version.go` - Application version data structures and variables.

internal/AGENTS.md

@@ -0,0 +1,42 @@
+# Preface
+
+This is the index file for the `internal/` directory's knowledge base. It provides context about the core internal packages of Chisel.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Overview
+
+The `internal/` directory houses the core business logic, components, and utilities of Chisel. It coordinates slice selection, dependency resolution, package fetching, file extraction, filesystem mutations, and manifest output. These packages are not part of Chisel's public API and must not be imported by external consumers.
+
+# Architecture
+
+Chisel's carving pipeline is coordinated by `slicer/`:
+
+```mermaid
+flowchart LR
+    CLI --> Release["Acquire and validate release"]
+    Release --> Select["Select and order slices"]
+    Select --> Fetch["Choose archives and fetch packages"]
+    Fetch --> Extract["Extract selected content"]
+    Extract --> Create["Create generated content"]
+    Create --> Mutate["Run ordered mutations"]
+    Mutate --> Manifest["Generate requested manifests"]
+```
+
+# Directory
+
+- `slicer/` - Main orchestrator for a Chisel run. Receives a slice selection, drives all other internal packages (setup, archive, cache, deb, fsutil, scripts, manifestutil) to completion, and writes the final filesystem and manifest.
+- `setup/` - Parses chisel-releases YAML definitions into the `Release` model and resolves slice dependencies, path conflicts, and package contention.
+- `deb/` - Debian package utilities: data tarball access, version comparison, and architecture handling.
+- `tarball/` - Extracts selected files from a package data tarball into a target directory.
+- `archive/` - Manages remote Ubuntu package archive sources over HTTP/HTTPS.
+- `cache/` - Content-addressable on-disk store keyed by SHA256 or SHA384 digest, with digest verification and last-use timestamps.
+- `fsutil/` - Core filesystem operations for writing files, directories, and symlinks into the target root filesystem.
+- `manifestutil/` - Generates the Chisel manifest.
+- `scripts/` - Executes Starlark mutation scripts defined in slice definitions.
+- `control/` - Parser for Debian control files (the metadata sections embedded in `.deb` archives).
+- `strdist/` - String distance and glob matching utilities. Implements a configurable edit-distance algorithm (`Distance`) with pluggable cost functions, and a `GlobPath` function that uses that algorithm to match file paths against patterns supporting `?`, `*`, and `**` wildcards.
+- `pgputil/` - Decodes and validates PGP signatures on package archive metadata. Wraps `golang.org/x/crypto/openpgp`.
+- `testutil/` - Shared test helpers used across unit tests: mock archive builders, composable content checkers, file presence and permission validators, tree dumpers, and permutation utilities.
+- `apacheutil/` - Apache-2.0-licensed slice-naming utilities shared across package boundaries.
+- `apachetestutil/` - Apache-2.0-licensed manifest test helpers used by tests in the `public/` packages.

public/AGENTS.md

@@ -0,0 +1,18 @@
+# Preface
+
+This document describes the scope of the `public/` directory, which contains Chisel's public API packages intended for consumption by external tools.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Overview
+
+The `public/` directory houses the two packages that form Chisel's stable public contract. They provide a generic sorted JSON-lines database format and the Chisel manifest schema built on it, enabling third-party tools such as SBOM generators and vulnerability scanners to consume Chisel output without depending on internal packages.
+
+# Important
+
+- When changing these packages or their dependencies, run `go run .github/scripts/external-packages-license-check.go` to verify the Apache-2.0 license boundary.
+
+# Directory
+
+- `jsonwall/` - Implements a generic database format with one JSON object per line, with fields and entries sorted for efficient search and iteration.
+- `manifest/` - Defines Chisel manifest schema 1.0 through the `Package`, `Slice`, `Path`, and `Content` record types, and uses `jsonwall` for serialization and deserialization. The carving pipeline applies Zstandard compression when generating manifest files.

snap/AGENTS.md

@@ -0,0 +1,13 @@
+# Preface
+
+This document describes the scope of the `snap/` directory within the Chisel repository, providing context for automated agents navigating and modifying the snapcraft configuration.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Overview
+
+The `snap/` directory contains the configuration necessary to build and package Chisel as a snap application. It defines the application metadata, execution confinement, and compilation steps for the `chisel` application using Go.
+
+# Directory
+
+- `snapcraft.yaml` - The primary manifest file defining the snap package. It configures strict confinement, utilizes the Go plugin, and outlines the build steps which depend on `cmd/mkversion.sh` for version injection.

tests/AGENTS.md

@@ -0,0 +1,26 @@
+# Preface
+
+This file serves as the local knowledge base index for the `tests/` directory. It outlines the scope and procedures for end-to-end integration testing within the Chisel repository.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Overview
+
+The `tests/` directory contains the Spread integration test suite. It exercises the `chisel` binary end to end, covering behavior that cannot be validated within individual Go packages.
+
+# Important
+
+- **Execution**: Integration tests are run with `spread`, not with `go test`. Each test scenario is a subdirectory containing a `task.yaml` with shell-based assertions.
+- **No build tags**: Unlike Go-based integration test suites, these tests require no `//go:build` directives. They are entirely shell-driven.
+- **Pre-built binary**: Spread compiles and provisions the `chisel` binary as part of the test environment setup defined in `spread.yaml` at the repository root.
+
+# Directory
+
+- `basic/` - Core slice extraction scenario verifying that files are correctly written to the target root filesystem and that mutation scripts are applied.
+- `find/` - Tests for the `chisel find` command, covering search by slice name and package.
+- `info/` - Tests for the `chisel info` command, verifying detailed slice metadata output.
+- `debug-check-release-archives/` - Tests detection of package-content conflicts that cannot be identified from slice definitions alone.
+- `pro-archives/` - Tests covering Ubuntu Pro subscription archive support (fips, fips-updates, esm-apps, esm-infra).
+- `use-a-custom-chisel-release/` - Tests the ability to override the default chisel-releases with a custom release tree.
+- `unmaintained/` - Tests for releases whose support window has ended.
+- `unstable/` - Tests for packages from unstable or unsupported releases.