Repository navigation
buck2_events: publish to a Bazel Build Event Service in OSS builds - #1599
jun-mercury wants to merge 3 commits into
Conversation
|
Hi @jun-mercury! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
9187e59 to
d09336d
Compare
|
@8Keep has imported this pull request. If you are a Meta employee, you can view this in D130203212. (Because this pull request was imported automatically, there will not be any future comments.) |
d09336d to
0a4b9d6
Compare
…crates Three new crates hold the generated Rust for the protocols an OSS build event service sink needs: - bes_grpc_proto: google.devtools.build.v1, the Build Event Service. - bazel_bep_proto: Bazel's build_event_stream and the messages it imports. - google_grpc_proto: google.api, google.rpc, google.longrunning and google.bytestream, with the serde helpers for Any. The Google protos move out of re_grpc_proto into google_grpc_proto, so the RE and BES crates share one copy. re_grpc_proto re-exports `google`, so code that names `re_grpc_proto::google::...` is unchanged. Nothing uses the new crates yet. Assisted-by: Claude:claude-opus-5-5
… stream needs Two changes to core crates that the OSS build event service sink builds on. Each is useful without it. RE requests carry the action's identity. ReActionIdentity gains the action id (the action digest), mnemonic (the action category), target id and configuration id, and the OSS client puts them in RequestMetadata. This covers executions, action cache lookups and writes, and uploads. Callers without an identity pass None. Servers such as BuildBuddy then show these per action. Events carry what a Bazel Build Event Protocol stream reports and Buck2's events left out: - an action's outputs (path, digest, size, kind) and output symlinks; - a test suite's labels and a test's timeout; - a materialization's URL; - `buck2 run`'s final command line (RunExecRequest); - fetches made by git-backed external cells (ExternalResourceFetch); - the repository root and working directory in the invocation metadata. Existing consumers see only new fields and new event variants. Assisted-by: Claude:claude-opus-5-5
An OSS build can stream its events to a server that speaks the Bazel Build Event Service protocol, such as BuildBuddy. That is the role Scribe plays in Meta's build. It is off unless [bes] is configured. The new sink code is compiled only under cfg(not(fbcode_build)), and the shared code in sink/scribe.rs and sink/remote.rs keeps the Scribe path under cfg(fbcode_build). - bazel_converter translates Buck2 events to Bazel BEP: the invocation and command line, targets, actions, tests, artifacts and the build graph. - bes_client streams them over PublishBuildToolEventStream. It retries from a bounded replay buffer that spills to disk, and writes inline files to the CAS off the event path. A refused stream is not redialled, and a stalled connection does not block a command. - The [bes] config takes an endpoint or the RE client's connection, plus headers with $VAR expansion. An unset variable turns BES off. It also takes a results URL, and TLS settings as the RE client has them. - The console prints the server's results URL. - The daemon finishes its streams before it exits. - `buck2 kill` finishes the streams before it starts the shutdown, after the kill's timeout is validated. While that drain runs (up to 5 s), the daemon still accepts new commands, because 9908473 made `DaemonShutdown::start_shutdown` the only place that marks the daemon as shutting down, and the drain has to finish before the shutdown starts. - Snapshots report where the sink's worker spends its time. docs/users/build_observability/event_system.md describes the configuration. Assisted-by: Claude:claude-opus-5-5
0a4b9d6 to
7a6ca25
Compare
Summary
Lets an OSS build stream its events to a server that speaks the Bazel Build Event Service protocol, such as BuildBuddy. That is the role Scribe plays in Meta's build.
The sink is off unless
[bes]is configured. The new sink code compiles only undercfg(not(fbcode_build)). The shared code insink/scribe.rsandsink/remote.rskeeps the Scribe path undercfg(fbcode_build), so fbcode builds are unchanged. Those two files are where a reviewer from the fbcode side will want to look.What it adds:
bazel_convertertranslates Buck2 events to Bazel BEP: the invocation and command line, targets, actions, tests, artifacts and the build graph.bes_clientstreams them overPublishBuildToolEventStream. It retries from a bounded replay buffer that spills to disk, and writes inline files to the CAS off the event path. A refused stream is not redialled, and a stalled connection doesn't block a command.[bes]takes an endpoint or the RE client's connection, plus headers with$VARexpansion; an unset variable turns BES off. It also takes a results URL, and TLS settings the same way the RE client does.docs/users/build_observability/event_system.mddescribes the configuration.This is part 3 of a stack of five that adds an OSS Build Event Service (BES) sink:
[bes]is configured, and compiled only in OSS builds. (this PR)buck2 kill --bes-drain-timeoutandbuck2 debug flush-bes. buck2: letbuck2 killwait for the BES server to acknowledge the daemon's events #1600Each PR targets main and contains the commits of the PRs before it. The commit this PR adds is the last one, d09336d. Each commit builds on its own.
Test plan
buck2 build //:buck2with a cargo-built buck2, the Build and Examples workflow's bootstrap step, passes on Linux, macOS and Windows with this stack and prelude: find libpython where each platform's sysconfig puts it #1606 to .github: get the Build and Examples workflow green on Linux, macOS and Windows #1609: https://github.com/MercuryTechnologies/buck2/actions/runs/42028461908. It caught thatbuck2_eventslistedprostandserde_jsononly for internal builds, which the BES sink uses in OSS builds, so buck2_events: publish to a Bazel Build Event Service in OSS builds #1599 now lists them for every build.cargo check --workspace --testspasses at each of the four commits.nightly-2026-08-15toolchain fromrust-toolchain.toml. Therust-overlaythatflake.lockpins predates that nightly, sonix developfails on main. I used a dev shell with a newerrust-overlay, and that change is not part of this stack.cargo test --libpasses for these crates: 841 tests, of which the BES subset (buck2_eventsandbuck2_re_configuration, filtered onsink::andbes) is 185.local_resource_registry::tests::protocol_release_uses_channel_not_pid_killfails on our NixOS test host because it spawns/bin/sleep, which that host lacks, and this stack leaves its file untouched.buck2 kill --bes-drain-timeoutandbuck2 debug flush-besran cleanly.Posted by Claude Code on behalf of Jun Matsushita.