Add workbench dashboards - #4128
Conversation
This is an initial schematization of datadog/grafana like dashboards for workbenches. This will serve a few purposes: 1. Give users more traditional integrated observability, leveraging our translation across backends 2. Store reliable source metrics to use in runs 3. Make it easier to generate useful monitors, which we will extend and make an event source for automated workbench execution.
Soffi AI SummaryThis PR introduces workbench dashboards — a Datadog/Grafana-style observability dashboard system embedded within AI workbenches. The feature enables users to compose grid-based dashboards with graphs sourced from any supported observability backend (Prometheus, Loki, Datadog, etc.), configurable variables/inputs, and flexible layout options. The implementation spans the full stack:
ChangesWorkbench dashboards
Updated: 2026-09-08 03:40 UTC |
Greptile SummaryThis PR introduces persistent workbench dashboards, typed graph and input definitions, GraphQL CRUD and datasource execution, structured observability label results, and monitor-trigger configuration. It also updates generated GraphQL clients and ReqLLM dependencies.
Confidence Score: 1/5This PR is not safe to merge until dashboard tool policies are enforced, monitor-triggered jobs consume the new monitor configuration, and monitor actor deletion semantics are defined. Dashboard queries can bypass configured tool restrictions, firing monitors do not successfully use their newly persisted prompt, modes, or actor, and monitor ownership can prevent user deletion. Files Needing Attention: lib/console/ai/workbench/toolchain.ex, lib/console/deployments/observability/dashboard.ex, lib/console/schema/monitor.ex, priv/repo/migrations/20260908020709_add_dashboard_schemas.exs
|
| Filename | Overview |
|---|---|
| lib/console/deployments/observability/dashboard.ex | Adds dashboard variable substitution and typed datasource execution, but forwards unconstrained caller values directly to observability tools. |
| lib/console/ai/workbench/toolchain.ex | Extends on-demand tool execution to workbenches, but invokes tools without enforcing configured workbench policies. |
| lib/console/schema/dashboard.ex | Defines dashboard graphs, inputs, layouts, and persistence with useful structural validation. |
| lib/console/schema/monitor.ex | Persists monitor prompt, modes, and actor identity, although those fields are not connected to triggered jobs. |
| lib/console/deployments/observability.ex | Adds dashboard CRUD and records monitor creators, with authorization checks on dashboard writes. |
| priv/repo/migrations/20260908020709_add_dashboard_schemas.exs | Adds aligned dashboard storage and monitor fields, but the monitor actor foreign key lacks deletion behavior. |
| lib/console/graphql/deployments/observability.ex | Adds the dashboard GraphQL contract and monitor execution fields consistently with generated schemas. |
| lib/console/graphql/resolvers/deployments/observability.ex | Adds dashboard CRUD, listing, and datasource resolvers while relying on the runtime for execution safeguards. |
Reviews (1): Last reviewed commit: "Add workbench dashboards" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| with tool when not is_nil(tool) <- Enum.find(tools, & Tool.name(&1) == name), | ||
| {:ok, %mod{} = t} when mod in @metrics_tools <- Tool.validate(tool, args) do | ||
| {:ok, %mod{} = t} <- Tool.validate(tool, args), | ||
| true <- mod in allowed do | ||
| mod.structured(t) |
There was a problem hiding this comment.
Dashboard datasource execution validates only the selected tool’s Ecto changeset before invoking it. The synthetic job lacks the workbench_id needed to load workbench policies, so those policies are never evaluated. Dashboard readers can therefore submit unrestricted substitutions and execute observability queries that the workbench policy was intended to reject.
How this was verified: The GraphQL input reaches direct tool invocation after unrestricted substitution, while the execution path neither loads nor evaluates the workbench policies.
Knowledge Base Used: Console AI workbenches
| field :prompt, :string | ||
|
|
||
| embeds_one :modes, Modes, on_replace: :update |
There was a problem hiding this comment.
The new monitor prompt, modes, and user identity are not propagated when a firing monitor starts an investigation. The alert consumer instead passes alert.workbench_webhook to create_workbench_bot_job/3. Monitor-created alerts do not set that association, so the value is nil and no matching function clause exists; the investigation crashes before creating a job. If a webhook is present, its prompt, actor, and modes are used instead of the monitor’s new fields.
Knowledge Base Used: Console AI workbenches
| alter table(:monitors) do | ||
| add :modes, :map | ||
| add :prompt, :string, limit: 2048 | ||
| add :user_id, references(:watchman_users, type: :uuid) |
There was a problem hiding this comment.
The new monitors.user_id foreign key uses PostgreSQL’s restrictive default deletion behavior. Because every newly created monitor records its creator, deleting that user through the existing user-deletion service fails with a foreign-key violation while any monitor references them. Similar persistent actor links define on_delete: :nilify_all or :delete_all; this reference also needs an explicit lifecycle policy.
Knowledge Base Used: Console backend platform
1 participant
This is an initial schematization of datadog/grafana like dashboards for workbenches. This will serve a few purposes:
Test Plan
Test environment: https://console.your-env.onplural.sh/
Checklist
Plural Flow: console