Skip to content

buck2: let buck2 kill wait for the BES server to acknowledge the daemon's events - #1600

Open
jun-mercury wants to merge 4 commits into
meta:mainfrom
MercuryTechnologies:bes-upstream/4-kill-drain
Open

jun-mercury wants to merge 4 commits into
meta:mainfrom
MercuryTechnologies:bes-upstream/4-kill-drain

Conversation

@jun-mercury

@jun-mercury jun-mercury commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

buck2 kill --bes-drain-timeout <duration> asks the daemon to deliver what its event sink still holds. It then waits, up to the timeout, until the server has acknowledged it, and only then exits. The result lists any invocation still unacknowledged.

buck2 debug flush-bes does the same for finished commands without stopping the daemon.

Without the flag, a kill behaves as before. A CI runner can use it so a job's last build reaches the server before the runner goes away. Our CI runs buck2 kill --bes-drain-timeout 600s at the end of each job.

This is part 4 of a stack of five that adds an OSS Build Event Service (BES) sink:

  1. Protocol crates: bes_grpc_proto, bazel_bep_proto and a shared google_grpc_proto. remote_execution: add the Build Event Service and Bazel BEP protocol crates #1597
  2. Core hooks: action identity in RE request metadata, and the event fields a BEP stream needs. buck2: name actions in RE request metadata, and record the data a BEP stream needs #1598
  3. The BES sink in buck2_events, off unless [bes] is configured, and compiled only in OSS builds. buck2_events: publish to a Bazel Build Event Service in OSS builds #1599
  4. buck2 kill --bes-drain-timeout and buck2 debug flush-bes. (this PR)
  5. A CI check of what buck2 sends a Build Event Service, in the Build and Examples workflow. examples/bes: check what buck2 sends a Build Event Service, in CI #1616

Each PR targets main and contains the commits of the PRs before it. The commit this PR adds is the last one, cc3929e. Each commit builds on its own.

Test plan

  • buck2 build //:buck2 with a cargo-built buck2, the Build and Examples workflow's bootstrap step, passes on Linux, macOS and Windows with this stack and prelude: find libpython where each platform's sysconfig puts it #1606 to .github: get the Build and Examples workflow green on Linux, macOS and Windows #1609: https://github.com/MercuryTechnologies/buck2/actions/runs/42028461908. It caught that buck2_events listed prost and serde_json only for internal builds, which the BES sink uses in OSS builds, so buck2_events: publish to a Bazel Build Event Service in OSS builds #1599 now lists them for every build.
  • cargo check --workspace --tests passes at each of the four commits.
    • I used the nightly-2026-08-15 toolchain from rust-toolchain.toml. The rust-overlay that flake.lock pins predates that nightly, so nix develop fails on main. I used a dev shell with a newer rust-overlay, and that change is not part of this stack.
  • At the top of the stack, cargo test --lib passes for these crates: 841 tests, of which the BES subset (buck2_events and buck2_re_configuration, filtered on sink:: and bes) is 185.
    • buck2_events, buck2_client_ctx, buck2_server
    • buck2_execute, buck2_execute_impl, buck2_build_api
    • buck2_re_configuration, buck2_external_cells, remote_execution
    • buck2_cli_proto, buck2_data
  • In buck2_test, 57 of 58 tests pass. local_resource_registry::tests::protocol_release_uses_channel_not_pid_kill fails on our NixOS test host because it spawns /bin/sleep, which that host lacks, and this stack leaves its file untouched.
  • We carry this sink in our fork, and it reports every build in our build lab to BuildBuddy. The stack is based on main at c7e71ec. A buck2 built from the head of the stack ran a scratch project against our BuildBuddy with remote execution on our workers. Builds, a failing build and a test run reported SUCCESS, BUILD_FAILURE and TESTS_FAILED. BuildBuddy recorded the test results in its test grid, and each remote execution carried its target and mnemonic. buck2 kill --bes-drain-timeout and buck2 debug flush-bes ran cleanly.

Posted by Claude Code on behalf of Jun Matsushita.

@meta-cla

meta-cla Bot commented Nov 4, 2026

Copy link
Copy Markdown

Hi @jun-mercury!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

@jun-mercury
jun-mercury force-pushed the bes-upstream/4-kill-drain branch 2 times, most recently from f0e087b to cc3929e Compare November 5, 2026 21:06
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 6, 2026
@meta-cla

meta-cla Bot commented Nov 6, 2026

Copy link
Copy Markdown

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@meta-codesync

meta-codesync Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

@8Keep has imported this pull request. If you are a Meta employee, you can view this in D130203222. (Because this pull request was imported automatically, there will not be any future comments.)

@jun-mercury
jun-mercury force-pushed the bes-upstream/4-kill-drain branch from cc3929e to 97724af Compare November 11, 2026 02:54
…crates

Three new crates hold the generated Rust for the protocols an OSS build event
service sink needs:

- bes_grpc_proto: google.devtools.build.v1, the Build Event Service.
- bazel_bep_proto: Bazel's build_event_stream and the messages it imports.
- google_grpc_proto: google.api, google.rpc, google.longrunning and
  google.bytestream, with the serde helpers for Any.

The Google protos move out of re_grpc_proto into google_grpc_proto, so the RE
and BES crates share one copy. re_grpc_proto re-exports `google`, so code
that names `re_grpc_proto::google::...` is unchanged.

Nothing uses the new crates yet.

Assisted-by: Claude:claude-opus-5-5
… stream needs

Two changes to core crates that the OSS build event service sink builds on.
Each is useful without it.

RE requests carry the action's identity. ReActionIdentity gains the action
id (the action digest), mnemonic (the action category), target id and
configuration id, and the OSS client puts them in RequestMetadata. This
covers executions, action cache lookups and writes, and uploads. Callers
without an identity pass None. Servers such as BuildBuddy then show these
per action.

Events carry what a Bazel Build Event Protocol stream reports and Buck2's
events left out:
- an action's outputs (path, digest, size, kind) and output symlinks;
- a test suite's labels and a test's timeout;
- a materialization's URL;
- `buck2 run`'s final command line (RunExecRequest);
- fetches made by git-backed external cells (ExternalResourceFetch);
- the repository root and working directory in the invocation metadata.

Existing consumers see only new fields and new event variants.

Assisted-by: Claude:claude-opus-5-5
An OSS build can stream its events to a server that speaks the Bazel Build
Event Service protocol, such as BuildBuddy. That is the role Scribe plays in
Meta's build. It is off unless [bes] is configured. The new sink code is
compiled only under cfg(not(fbcode_build)), and the shared code in
sink/scribe.rs and sink/remote.rs keeps the Scribe path under
cfg(fbcode_build).

- bazel_converter translates Buck2 events to Bazel BEP: the invocation and
  command line, targets, actions, tests, artifacts and the build graph.
- bes_client streams them over PublishBuildToolEventStream. It retries from a
  bounded replay buffer that spills to disk, and writes inline files to the
  CAS off the event path. A refused stream is not redialled, and a stalled
  connection does not block a command.
- The [bes] config takes an endpoint or the RE client's connection, plus
  headers with $VAR expansion. An unset variable turns BES off. It also
  takes a results URL, and TLS settings as the RE client has them.
- The console prints the server's results URL.
- The daemon finishes its streams before it exits.
- `buck2 kill` finishes the streams before it starts the shutdown, after
  the kill's timeout is validated. While that drain runs (up to 5 s), the
  daemon still accepts new commands, because 9908473 made
  `DaemonShutdown::start_shutdown` the only place that marks the daemon as
  shutting down, and the drain has to finish before the shutdown starts.
- Snapshots report where the sink's worker spends its time.

docs/users/build_observability/event_system.md describes the configuration.

Assisted-by: Claude:claude-opus-5-5
…emon's events

`buck2 kill --bes-drain-timeout <duration>` asks the daemon to deliver what
its event sink still holds. It then waits, up to the timeout, until the
server has acknowledged it, and only then exits. The result names any
invocation still unacknowledged.

`buck2 debug flush-bes` does the same for finished commands without stopping
the daemon.

The daemon drains once, for the whole process, before any tenant is evicted,
so a kill under daemon sharing drains the shared sink before the evictions
start. The client waits for the drain on top of its usual bounds. Evicting
one tenant (`kill_unidaemon_tenant`, the tenant proxy) does not drain.

Without the flag a kill behaves as before. A CI runner can use it so a job's
last build reaches the server before the runner goes away.

Assisted-by: Claude:claude-opus-5-5
@jun-mercury
jun-mercury force-pushed the bes-upstream/4-kill-drain branch from 97724af to f68f69d Compare November 17, 2026 01:26

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant