Repository navigation
Conversation
Reads (get/exists/zrange_by_score/scan/zscan/bf_exists) in both the KV-store and cache-backend Redis clients now retry up to 2 times with capped exponential backoff (10ms/20ms, max 50ms) on top of the existing connection-manager reconnect. Kept deliberately small and bounded to avoid reintroducing the pile-up behavior behind the 2026-07-09 congestion-collapse incident. Mutating operations (incr/set/delete/ expire/clean_index) are left untouched since a lost-response retry could double-apply them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
feat(stats): refactor stats handling to include caller IP
feat(stats): add cached read tracking to ReadStats and MeteredStore
…nce error handling
Add bounded retry with backoff for Redis read operations
…server Remove the 'metrics' cargo feature and the prometheus/lazy_static dependencies from the public fastedge-lib crates: - runtime::util::metrics is now a backend-agnostic hook: embedders install a process-global sink via set_sink(); with none installed (fastedge-run CLI) reporting is a no-op - runtime::instances keeps live/peak counts in plain atomics; flush_peak() now returns the drained peak instead of setting a gauge - AppResult variants and http-service call sites are no longer feature-gated The Prometheus implementation (fastedge_call_count, fastedge_error_total_count, fastedge_request_duration, fastedge_wasm_memory_used, fastedge_wasm_instances_live/peak) moves to server/src/metrics.rs; the sink is installed at startup and the instance gauges are set on each scrape (same semantics). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ve consistency in `runtime::instances`
Expose the four missing key-value read metrics on the Prometheus scrape:
- fastedge_kv_reads_total{result="miss"|"hit"} splits reads into direct
(backing-store) and in-process cache hits. Commands that are never
cached (scan/zscan/zrange_by_score/bf_exists) count as misses.
- fastedge_redis_read_retries_total{store,command} counts retried read
attempts only, so rate() gives retries since the previous scrape.
- fastedge_redis_read_duration_seconds{store,command} times uncached
reads; the average since the previous scrape is rate(_sum)/rate(_count).
Cache hits are excluded (in-process lookup) and failed reads are not
observed, since their timeout-dominated duration would skew the
latency of served reads.
The key-value store lives in the fastedge-lib workspace and must stay
free of a metrics backend, so it reports retries through a new
set_read_retry_observer hook that the server installs at startup.
All new series are materialised at 0 by redis_metrics::init() and are
documented with PromQL in docs/METRICS.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the guest trapped before invoking `response-outparam::set`, the
error was rebuilt with `bail!("...: {e:?}")`, which flattens the cause
into a string and drops the source chain. `map_err`/`fail_reason_of`
classify errors by downcasting `root_cause()` to `wasmtime::Trap` or
`Elapsed`, so every such failure fell through to the generic branch and
returned 530/INTERNAL_STATUS_EXECUTE_ERROR instead of 532/3010 - even
though the stats row was correctly recorded as TIMEOUT.
Use `.context()` to attach the message while keeping the chain intact.
Also stop logging the same backtrace twice: the spawned task logged
"incoming handler" and the error was then logged again as "execute"
after propagating up. The task-side log now only fires when the error
cannot propagate (response headers already flushed), which was the only
case it was needed for.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lls_total` Replace `fastedge_call_count` and `fastedge_error_total_count` with a single `fastedge_calls_total` metric that tracks all app call outcomes under an `outcome` label. Add support for the new `SUSPENDED` app result. Update HTTP service to emit metrics for requests with `DISABLED`, `RATE_LIMITED`, and `SUSPENDED` statuses. Add unit test coverage to ensure correct metric allocation across outcomes.
fix(http-service): preserve error chain so wasi-http timeouts return 532
…metrics Epx 298 proxywasm accept backlog metrics
…backend integration
- Added `fastedge_kv_cache_bytes`, `_entries`, and `_max_bytes` metrics for tracking in-process key-value store cache occupancy. - Implemented lifecycle management to dynamically remove metrics for idle/disconnected stores. - Updated configuration to accurately define default cache sizes in bytes and ensured compatibility with Prometheus scraping. - Enhanced test coverage for cache behavior, including capacity, eviction, oversized values, and label sanitization.
…_id and metric_label
…ations - Enhanced error classification for Redis read failures with clearer differentiation between transient and permanent errors. - Introduced `retry_backoff`, `is_guest_fault`, and `read_error` helpers to streamline retry logic. - Optimized retry mechanism with capped backoff for `WAIT_AND_RETRY`. - Improved guest-facing error messages for deterministic errors. - Updated unit tests to cover new error handling logic comprehensively. - Integrated `read_status_of` in ProxyWasm host to map errors to appropriate status codes.
- Updated default cache TTL from 300 seconds to 60 seconds to enable quicker expiration.
…andling Add `epoch_grace` module to refine epoch timeout decisions by distinguishing kernel-parked threads from CPU-exhausted guests. Track grace outcomes with the new `fastedge_epoch_stall_grace_total` metric. Update store functionality to utilize CPU-aware deadlines, enhancing reliability under memory pressure.
…upport Enhance error classification during app initialization (`_initialize`, `on_context_create`) to distinguish between initialization-phase timeouts and request-phase timeouts. Introduce new `CallPhase` enum to provide phase-specific context for metrics. Update metrics labels to differentiate `init_timeout` from standard `timeout`.
Implement `rebase` functionality to redefine baseline for credited deadline extensions, enhancing CPU usage accuracy while maintaining stall grant allowances. Introduce and refine supporting tests for the new mechanism.
Remove the `epoch_grace` module and its related `fastedge_epoch_stall_grace_total` metric. Eliminate CPU-aware grace handling and its integration in the runtime. Update tests, documentation, and metrics initialization accordingly.
Port the access-log work from feat/integration-with-log-sender onto current main (that branch is ~210 commits behind and carries the FastEdge-lib subtree at lib/ instead of fastedge-lib/). Emit exactly one record per processed request — for every outcome, including early rejections — describing how the edge server handled it: traceparent, final status, X-CDN-Internal-Status, app/client ids, method, uri and execution time. Transport is abstracted behind `runtime::access_log::AccessLogSender`; the server implements it as best-effort RFC 3164 syslog datagrams over UDP to the trace stream (default 127.0.0.1:9514, `--access-log-addr` / `--access-log-role`). Init failure falls back to the no-op sink. Covered paths: - ProxyWasm (v1/v2/v2a/v2b/v2c/v2d/v3), one record per lifecycle callback, tagged with the trigger and the initiating service. - Preview2 http-handler and standard wasi:http. - Outbound requests (`proxy_http_call` and the WASI/http-backend fetch client) as their own `fetch` records, correlated by traceparent via a dependency-free `ExtRequestLog` seam in http-backend. Requests without an inbound `traceparent` get a freshly generated W3C id (`runtime::trace::new_traceparent`) instead of a nanoid, and the resolved id is pinned onto the request so outbound records join the inbound one. `traceparent` is added to the default propagate-headers set. nanoid is dropped from proxywasm and http-service. Deviations from the source branch, for current main: - Initiator/version mapping extended to V2d (core-proxy) and V3 (nginx), matching the connection-gauge classification. - ProxyWasm record duration reads the per-call `Instant` already used by the metrics path (ungated from the `metrics` feature) rather than the stats row, whose `get_time_elapsed()` is a request-level accumulator and would over-report on streaming body callbacks. - The cdn-config/LMDB commit on that branch is a separate feature and is not included.
- Upgraded to Wasmtime 48.0.0, adjusting dependencies and addressing breaking changes. - Refactored `WasiHttpHooks` implementation to align with Wasmtime 48's new `HttpHooks` design. - Streamlined epoch-related bookkeeping for HTTP requests and improved resource table handling. - Updated `Cargo.lock` with new versions and removed unused or outdated dependencies.
- Introduced `EpochRefund`, `ResponseBodyEpochRefund`, and `RequestBodyEpochRefund` to improve epoch pause accounting for network wait times during HTTP request and response processing. - Enhanced `WasiHttpHooks` to ensure accurate epoch refunds for guest blocking on network I/O. - Updated `Cargo.toml` and `Cargo.lock` to include `http-body` dependency.
…cy and locked builds - Updated `Cargo.lock` with new `wasmtime` commit references. - Adjusted GitHub Actions workflow to include `--locked` flag for `cargo deb` commands.
…gement - Updated `update-preview1.sh` for improved Wasmtime version tracking and script reliability. - Updated `Cargo.lock` to include Wasmtime 48.0.2 and dependency adjustments. - Refactored references to dependencies for consistency and removed outdated sources.
feat(runtime): upgrade Wasmtime to 48.0.0 and refactor HTTP hooks
The process-wide cache keyed a graph by backend + canonical directory, so a model replaced in place (rsync/cp over the same files) was never seen: every executor rebuild hit the cached graph and `mark_used` kept it alive as long as there was traffic, serving the old model until restart. On `main` the per-executor registry re-read the files on each rebuild. Each cache entry now carries a `Revision` of the directory's files (name, size, mtime — one `stat` per file, no hashing of the weights). Preload compares it under moka's per-key `and_try_compute_with`: same revision is a hit, a different one replaces the entry atomically, so the new model is loaded at the next executor rebuild and the old graph is freed once the executors built on it are gone — no second copy lingering for the 4 h idle timeout. Symlink swaps were already covered by canonicalisation. Loads are rare and cost seconds and gigabytes, so each one is logged at info: backend, path, guest name, bytes on disk, RSS delta (Linux), load time and whether it replaced a stale revision. Evictions log their cause (idle, replaced, capacity); cache hits log at debug.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Graph-cache correctness, Redis error exposure, configuration compatibility, and incomplete access-log accounting must be addressed.
10 open findings
What changed in this PR
Upgrades the runtime to Wasmtime 48 and introduces process-wide WASI-NN graph caching, alongside broader observability and reliability changes.
Changes:
- Preloads and shares WASI-NN graphs across engines and stores.
- Migrates WASI/WASI-HTTP integrations to Wasmtime 48.
- Adds access logging, metrics hooks, Redis retries, and KV cache tracking.
| File | Description |
|---|---|
src/main.rs |
Initializes the shared graph registry. |
src/key_value.rs |
Updates the stats stub. |
src/context.rs |
Preloads graphs on a blocking worker. |
crates/runtime/src/util/stats.rs |
Adds cached-read and discard hooks. |
crates/runtime/src/util/mod.rs |
Makes metrics unconditional. |
crates/runtime/src/util/metrics.rs |
Replaces Prometheus with a sink API. |
crates/runtime/src/trace.rs |
Generates W3C trace identifiers. |
crates/runtime/src/store.rs |
Integrates preloaded graphs and HTTP hooks. |
crates/runtime/src/service.rs |
Shares the graph registry with engines. |
crates/runtime/src/registry.rs |
Implements graph loading and caching. |
crates/runtime/src/logger.rs |
Removes obsolete WASI file support. |
crates/runtime/src/limiter.rs |
Adapts Wasmtime error types. |
crates/runtime/src/lib.rs |
Migrates runtime APIs and HTTP hooks. |
crates/runtime/src/instances.rs |
Reworks instance metrics as atomics. |
crates/runtime/src/app.rs |
Changes KV cache-size semantics. |
crates/runtime/src/access_log.rs |
Defines inbound request logs. |
crates/runtime/Cargo.toml |
Updates runtime dependencies and features. |
crates/key-value-store/src/redis_impl.rs |
Adds Redis read retries and classification. |
crates/key-value-store/src/lib.rs |
Adds cache outcomes and retry observers. |
crates/key-value-store/Cargo.toml |
Enables Tokio for Redis retries. |
crates/http-service/src/state.rs |
Exposes outbound-log handles. |
crates/http-service/src/lib.rs |
Adds access logs and request accounting. |
crates/http-service/src/executor/wasi_http.rs |
Migrates the WASI-HTTP executor. |
crates/http-service/src/executor/mod.rs |
Updates body imports. |
crates/http-service/src/executor/http.rs |
Adapts HTTP execution to new APIs. |
crates/http-service/Cargo.toml |
Removes obsolete metrics/WASI dependencies. |
crates/http-backend/src/lib.rs |
Logs outbound requests. |
crates/http-backend/src/access_log.rs |
Defines the outbound-log seam. |
crates/cache/src/lib.rs |
Exposes direct cache operations. |
CLAUDE.md |
Documents the new runtime architecture. |
Cargo.toml |
Upgrades Wasmtime to release 48. |
Cargo.lock |
Locks upgraded dependencies. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+84
to
+85
| fn is_guest_fault(error: &RedisError) -> bool { | ||
| matches!(error.retry_method(), RetryMethod::NoRetry) && error.code().is_some() |
Comment on lines
77
to
78
| SUCCESS, | ||
| #[cfg(feature = "metrics")] | ||
| UNKNOWN, |
Comment on lines
+479
to
+482
| // Capture the outbound method/target for the access-log record before | ||
| // `make_request` consumes the request. | ||
| let log_method = method_str(&req.method); | ||
| let log_uri = req.uri.to_smolstr(); |
Comment on lines
+230
to
+234
| tokio::task::spawn( | ||
| async move { | ||
| match task.await { | ||
| Ok(Ok(())) => {} | ||
| Ok(Err(e)) => tracing::warn!(cause=?e, "incoming handler"), |
Comment on lines
+399
to
+401
| log.app_id = Some(cfg.app_id); | ||
| log.client_id = Some(cfg.client_id); | ||
| log.app_name = Some(app_name.to_string()); |
Comment on lines
+47
to
52
| /// Maximum **bytes** held by this store's in-process value cache (keys, | ||
| /// values and per-entry overhead), not a number of entries: entry count | ||
| /// cannot bound memory, since the same 25 000 entries are 2 MiB of | ||
| /// 64-byte values or 200 MiB of 8 KiB ones. | ||
| #[serde(default = "KvStoreOption::default_cache_size")] | ||
| pub cache_size: u64, |
Comment on lines
+401
to
+405
| let head = match self.inner.backend_request(head) { | ||
| Ok(head) => head, | ||
| Err(e) => { | ||
| tracing::warn!(cause=?e, "backend request"); | ||
| let cause = e.to_string(); |
Comment on lines
+444
to
+451
| // One access-log record per outbound request (status 0 = no response | ||
| // received, e.g. connection refused / timeout). | ||
| if let Some(handle) = log_handle { | ||
| let status = match &sent { | ||
| Ok((incoming, _)) => incoming.status().as_u16(), | ||
| Err(_) => 0, | ||
| }; | ||
| handle.log(&log_method, &log_uri, status, elapsed); |
Comment on lines
+99
to
+101
| entry.file_name().hash(&mut hasher); | ||
| meta.len().hash(&mut hasher); | ||
| meta.modified().ok().hash(&mut hasher); |
Comment on lines
+269
to
+272
| pub fn mark_used(&self, graphs: &[LoadedGraph]) { | ||
| for g in graphs { | ||
| let _ = self.0.get(&g.key); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.