Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@
Thank you for contributing to GitHub CLI!

If you are proposing a fix for a security issue, STOP and follow .github/SECURITY.md instead.

Keep the entire pull request self-contained, reviewer-facing, and diegetic: describe the
change, rationale, and evidence within the context of the repository and pull request. Do not
narrate how the pull request was produced or refer to private conversations, prior agent work,
or other behind-the-scenes context. Omit those details unless they materially affect review;
if they do, state the relevant fact and its significance directly.
-->

<!-- List related issues here. Use `fixes` or `closes` keywords to associate an issue number. -->
Expand Down
4 changes: 4 additions & 0 deletions .github/skills/cli-exercise/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
node_modules/
/tool/cli-exercise
/tool/cli-exercise.exe
/tool/bin/
147 changes: 147 additions & 0 deletions .github/skills/cli-exercise/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
---
name: cli-exercise
description: Exercises real command-line programs from explicit cases or a goal, with faithful terminal recordings and observed results. Use when asked to record CLI test cases, demonstrate command behavior, validate an interactive workflow, or explore a command against a goal; not for ordinary unit-test execution or skill authoring.
license: MIT
compatibility: Requires Go 1.27+ to build the helper, Node with Tuistory, ffmpeg/ffprobe, and an installed monospaced font. Builds start offline; ask before downloading missing dependencies.
---

# CLI Exercise

Exercise the caller's selected executable and return local evidence. Accept
precise cases, a broad goal, or a mixture. Preserve explicit requirements while
letting the agent choose useful investigations where the caller left room.

By default, produce one chaptered session video, even for one case. Each case
covers one behavior and has a very short natural-language behavior/outcome title.
Actually execute and record that case's preparation, one exercise invocation,
then validation before starting the next case. Approved cleanup can follow
validation. Never batch all setups or validations or reorder historical clips
to imply that execution order. Exact supplied steps and narrower coverage remain
authoritative: ask about conflicts rather than silently adding, splitting, or
reordering a caller's case.

Deliver media only by default. Infer clear output intent: a video exercising a
PR calls for an MP4, while a request for an interactive review report calls for
media plus HTML. Ask only when the requested deliverable is ambiguous. Generate
HTML with `--html` only when wanted; do not create or open it for video-only
requests. Keep raw captures and JSON results as internal evidence.

The internal `SKILL.md` files below are bundled instruction modules. Read the
needed module from this package and follow it in the current agent context.
They do not require new agents, sessions, or separately installed sibling skills.

## Load only what is needed

This entry file is the routing overview. Load the selected flow, not every
internal module. Execute the Go helper as a tool; read its source only when
diagnosing a problem or reviewing the implementation.

Go owns prerequisite checks, execution policy, controllers, capture,
verification, and rendering. Node is only a mechanical adapter to Tuistory's PTY
API. The skill does not require another scripting-language environment.

| Need | Resource to read |
|---|---|
| Check readiness or request installation | [Preflight flow](internal/cli-exercise-preflight/SKILL.md) |
| Preserve supplied cases | [Exact-case flow](internal/cli-exercise-exact/SKILL.md) |
| Decide how to pursue a goal | [Exploration flow](internal/cli-exercise-explore/SKILL.md) |
| Start or drive the terminal | [Run flow](internal/cli-exercise-run/SKILL.md) |
| Verify, render, or clean up | [Evidence flow](internal/cli-exercise-evidence/SKILL.md) |
| Normalize the caller's cases | [Run-contract reference](references/run-contract.md) |
| Call the bundled helpers | [Runtime interface](references/runtime-interface.md) |
| Coordinate cases and assemble one video | [Recording sessions](references/recording-sessions.md) |
| Author a deterministic controller | [Controller reference](references/controllers.md) |
| Resolve missing prerequisites | [Prerequisite reference](references/prerequisites.md) |
| Inspect representative routing behavior | [Input examples](references/examples.md) |
| Maintain or evaluate the package | [Maintenance reference](references/maintenance.md) |
| Run an authoring evaluation | [Agent-behavior rubrics](tests/skill-evaluations.json) |

These direct links keep resources one hop from the entry point. For an exact
request, do not load the exploration flow unless the caller also supplied an
exploratory case. Do not load controller details for a live-only run.

## 1. Preserve the request and choose the fidelity policy

Classify each case, not just the entire request:

- **Exact:** the caller supplied cases or required steps. Preserve commands,
arguments, input values, order, assertions, and timing/output constraints.
Do not add exploratory steps or silently substitute an equivalent-looking
operation for a specified key sequence.
- **Explore:** the caller supplied a goal or expected outcome. Choose checks
and interactions within the agreed context and permitted effects. Separate
supplied expectations from your hypotheses.

Input fidelity is independent of execution strategy. Either policy can use a
bounded controller, live observation/action calls, or both. Switching strategy
never grants permission to change an exact case.

Resolve only missing details that affect correctness or authority:

- The executable or build to exercise, with its actual path and identity.
- Cases or goal, supplied context, expected outcomes, and initial state.
- Permitted files, hosts, accounts, repositories, and other effects.
- Desired media format and constraints. Honor an explicit format; use GIF for
an unspecified terminal animation and MP4 when the caller requests video.

Default to condensed presentation, with static pauses shortened. Honor explicit
real-time requests and retain the original capture timestamps. Keep timing
adjustments in the recording metadata, not repeated on-screen labels.

A request such as "upload an attachment" is not permission to choose an
arbitrary account or publish to the current repository. Ask for the missing
target/authorization before performing a write. Prefer synthetic data and
isolated configuration.

Command discovery is a separate case property. A case may provide the command
and test its use, or explicitly require finding it. Do not silently turn a
known-command case into a discoverability test.

Preserve the non-secret original request alongside normalized cases. Never copy
credentials into the contract or recording; use approved secret references or
report blocked if the requested literal recording would disclose them. Use
[the run contract](references/run-contract.md); unsupported or ambiguous
requirements must be clarified or marked blocked, not discarded.

## 2. Establish readiness and authorization

Read [the preflight module](internal/cli-exercise-preflight/SKILL.md).
It checks dependencies. If setup is needed, propose commands using the pinned
requirements, obtain caller approval, and recheck afterward.

Installation approval and permission to modify test resources are independent.
Record the executable path, version/revision, checksum, accepted case contract,
and approved effect scope before starting the subject.

Use a secure private workspace outside the repository. Do not put recordings,
credentials, runtime caches, or operator configuration in this skill directory.
No secret values belong in the contract or a tool request.

## 3. Load the appropriate internal flow

- Supplied cases: [record exact cases](internal/cli-exercise-exact/SKILL.md).
- Broad goals: [explore a goal](internal/cli-exercise-explore/SKILL.md).

Both use [the shared run module](internal/cli-exercise-run/SKILL.md).
It provides real terminal observation, input, controller execution, recording,
and bounded process ownership. Read only the internal modules needed.
Use [a recording session](references/recording-sessions.md) to keep case and phase
order, carry approved state between related commands, and assemble their actual
recordings with `evidence --session`, not the per-invocation `--run-dir` path.
Keep validation commands in that recording. The session overview uses the
task-specific title, derived fidelity, and case results, with 10 seconds per
readable page. Progress spans the entire final video.

## 4. Verify, render, and clean up

Always finish through [the evidence module](internal/cli-exercise-evidence/SKILL.md),
including when the case is blocked or the subject fails.

Return the case outcome, meaningful observed behavior, requested media paths,
executable identity, inspection coverage, and limitations. Keep case failure,
capture failure, and render failure distinct. A video is evidence of an observed
flow, not proof of general correctness or a replacement for tests.

Include a report link only when the caller requested the report. Do not leave
a host, terminal, recorder, or playback server running. Publishing the evidence
is a separate, explicit action.
140 changes: 140 additions & 0 deletions .github/skills/cli-exercise/internal/cli-exercise-evidence/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
---
name: cli-exercise-evidence
description: Internal evidence, rendering, and cleanup flow for cli-exercise. Verifies expectations and returns local artifacts without publishing by default.
user-invocable: false
license: MIT
---

# Evidence and cleanup

Read the immutable contract, runtime result, input/controller trace, and recorded
states. Keep the caller's case outcome separate from capture and rendering status.
For the default complete recording, follow [recording sessions](../../references/recording-sessions.md)
and assemble all requested cases and phases with `evidence --session`, even for
one case. `--run-dir` remains useful for a per-invocation clip, not the normal
session deliverable.

## Verify the requested claim

Check every expectation against actual evidence. Built-in terminal assertions
do not prove separate stdout/stderr routing or remote resource state.
Complete manual expectations with approved independent observations, or mark
them blocked. Never infer success solely from exit zero or a produced video.
Record any case-validation commands as session chapters rather than leaving
their execution outside the video. Execute them immediately after that case's
exercise, before the next case, not as a final batch when assembling evidence.
Honor exact supplied steps and explicit narrower coverage; ask about conflicts
rather than adding validation commands the caller excluded.

For writes, verify the object actually written rather than an unrelated or
pre-existing object. Preserve failures and partial-effect facts. Do not repeat
a potentially successful write just because readback failed.

For exact cases, confirm that the supplied steps, values, order, and assertions
were preserved. For exploration, distinguish supplied outcomes from inferred
checks and record meaningful changes of approach.

Judge pass/fail against expectations, not exit-code positivity. An expected
nonzero exit can pass; an exit-zero assertion failure cannot. Keep `observed`,
`blocked`, and unknown results visibly distinct from `passed`.

## Render without fabricating output

Use the Go helper's `evidence` command with the skill root, session manifest,
and ready receipt. Honor the requested GIF/MP4 formats and size constraints.
Condensed presentation is the default when timing is unspecified. Honor explicit real-time requests.
Retain source timing and record shortened pauses and reading holds in the
rendering metadata. Do not add repeated timing labels to the video.
Select an existing compatible primary font, pass it with `--font FILE`, and let
Go rasterize it; FFmpeg encodes the resulting frames.
Do not add an interpreter, browser, or font service as an unapproved fallback.

For another output from an existing capture, use `--format` and/or `--timing`
on the evidence command. Condensed output does not require a separate approval.
Each invocation writes a new rendering and records its presentation options
separately. Do not rewrite
the execution contract or rerun a command merely to change its presentation.

Every rendering requires `--font FILE`. If it reports a missing glyph, repeat
`--font-fallback FILE` to define the complete ordered fallback chain. These
options apply to session chapters and the overview too.
Select existing compatible fonts, then rerun only `evidence`; the new report
records the rendering choices and font hashes without changing the capture.

The default deliverable is the requested media, without HTML. Add `--html` only
for an interactive report request. This applies to both standalone and session
renders, including per-chapter reports. Raw capture, JSON results, and inspection
data remain available regardless of that choice.

For an annotated session, require:

- An opening summary with the task-specific manifest title, fidelity derived
from exercise contracts (`Exact run`, `Exploration`, or `Mixed run`), and brief
case-title/result rows. Give every readable page 10 seconds. Paginate long
lists; never drop cases or shrink them into unreadable text.
- Visible phase labels: purple `Preparation`, blue `Running`, and `Validation`
in green only for passed checks or red for failed checks. Blocked, observed,
and unknown validation results need explicit non-passing labels and styling.
- A thin progress line spanning the whole final video, including overview pages,
reading holds, and case boundaries, without restarting for each clip.
- The actual recorded invocation at the top of the terminal area, followed by
captured output. Present the immutable executable and arguments, not a
fabricated keystroke transcript or an extra shell execution to print them.
- A separate lower panel explaining the behavior or check in plain language,
not `Command: ...`. Use run titles and recorded notes; keep product output
intact and unobscured.

An explicit unannotated real-time request retains no-overlay behavior: do not
add an overview, phase labels, command header, progress line, or explanation
panel. Do not add condensed reading holds.

The terminal picture must come from recorded real states. Never cover, replace,
paste, or retrospectively redact product output. Do not reorder captures to
make historical execution appear sequential. Nonchronological, overlapping,
or reused captures must not be presented as a valid sequential session.
Legacy captures missing runtime timestamps remain readable only with an
explicit `execution order unverified` warning.

If private input caused unsafe output, stop and fix the input at its source.
A new take must respect the original case and any remaining authorization for
effects. Do not repeat remote writes without resolving their existing outcome.

## Inspect before handoff

Inspect native-resolution first/final frames and relevant transient states.
Check readable monospaced text, wrapping, editor/cursor behavior, captions,
timing, and privacy. Inspect every overview page, its 10-second duration, case
coverage, fidelity label, outcome styling, invocation display, and progress at
holds and chapter boundaries. Confirm no overlay appears in an unannotated cut.
Inspect every distinct visual state for a privacy or
transient-state claim. The helper saves identical pixel images once and retains
their frame, timestamp, source-state, and revision mappings in the manifest.
Use those mappings to locate repeated appearances rather than expecting a
separate PNG for each frame.
State actual coverage. Decoding every frame is not the same as visually
reviewing every frame.
Inspect the `encodedSamples` from each requested format as well as the
composited/source images. The latter alone cannot reveal encoding artifacts.

Record executable identity, contract/controller hashes, dimensions, duration,
frame counts when available, and any limitation on what the evidence proves.
Use each format's observed media metadata, including GIF's centisecond timing
granularity, rather than treating the planned frame rate as exact media timing.

The generated report records visual review as pending at rendering time. After
inspection, write a separate `visual-review.json` beside that report, bound to
its contract hash and `inspection.manifestSha256`. List the actual image paths
reviewed, coverage, findings, and the review outcome. Do not rewrite raw capture
or relabel sampled coverage as exhaustive. Link this review receipt at handoff.

## Close and return

Confirm that owned runtime, recorder, and transport processes are stopped.
Do not stop other terminal sessions or globally shut down another user's daemon.
Keep contracts, results, traces, and final artifacts in the private workspace.
Clean up only known temporary data owned by the run.

Return the requested local media paths and concise case outcome. Include HTML
only when requested; it must work without a standing playback server. Publishing
evidence or modifying a PR is separate and requires the caller's request and
appropriate approval.
Loading
Loading