Skip to content

feat(logs-page): add a Logs page for browsing and fetching vehicle logs - #109

Merged
dakejahl merged 6 commits into
mainfrom
feat/logs-page
Aug 6, 2026
Merged

feat(logs-page): add a Logs page for browsing and fetching vehicle logs#109
dakejahl merged 6 commits into
mainfrom
feat/logs-page

Conversation

@dakejahl

@dakejahl dakejahl commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a Logs page listing every log logloader knows about — on the vehicle, downloaded, uploaded — and lets the operator pick which ones to fetch and publish. Uploaded logs link straight to the plot in the onboard Flight Review.

Depends on ARK-Electronics/logloader#32, which the submodule bump points at; merge that first.

Problem

logloader decided on its own what to download and upload, and there was no way to see what it had or to ask for a particular log. Its counterpart change makes fetching request-driven, which needs somewhere to make requests from.

Two things surfaced while wiring it up that are not specific to this page. The dev server gzips responses by default and its compression middleware buffers until it has enough bytes, which for an SSE stream is never — measured through npm run serve with a browser's Accept-Encoding: 0 bytes and an EventSource stuck at readyState 0, against 3332 bytes through the gateway. That has been silently breaking the journal viewer, the network-stats stream and the firmware-progress stream in dev too. Separately, http-proxy-middleware v3 reads handlers from on, so the v2 onError/logLevel options every proxy block uses are dead: a stopped upstream answers with the default plain-text 504 rather than the JSON the UI parses.

Upgrading an existing install was also unsafe. merge_configs.py takes its structure from the new template and prunes scalars the template lacks — right for a removed field, wrong for a renamed one. Simulating an upgrade of a real logloader.toml produced [upload_remote] enabled = false and the stock url, so a vehicle configured to upload to a remote Flight Review would silently stop.

Solution

logloader serves a localhost API on :3005; the gateway proxies it at /api/logloader, including the SSE stream that drives live transfer progress. The page follows the house patterns: Options API, an axios service module, an inline EventSource with named events, and the SystemPage loading/error/retry triad.

Select all or a subset, then Download & Upload, Download Only, Upload, Cancel — or Refresh: logloader no longer polls the vehicle at all, so the page asks for a fresh listing when it loads and on demand, and a "Logger running" chip mirrors MAV_SYS_STATUS_LOGGING so it is visible why an automatic fetch is waiting. A log the vehicle is still writing never appears, so it cannot be selected at a size it will not keep. An upload records only the relative path Flight Review redirects to, so targets carry their base url and a remote plot link resolves against the right origin rather than the ARK-OS host.

compress: false on the dev server, on: { error } for all five proxies, and migrate_logloader_config.py run from postinst before the merge so a rename reaches merge_configs as an ordinary same-shape merge.

Review of the wiring found three more, all fixed here: supplying on.error in http-proxy-middleware v3 replaces its error plugin and its guards, so an upstream dying mid-SSE made writeHead throw on a response whose headers were long sent — an uncaught exception in a socket callback that took the whole gateway down; EventSource only retries network failures, so the gateway's 502 while the daemon restarts closed the stream for good and the page said "reconnecting" forever; and the migrated download.use_burst was absent from the new config template, so merge_configs pruned the very key the migration had just preserved.

logloader's config tables are one level deep because the shared TomlEditor renders exactly one level; with two, every upload setting silently vanished from the Services page config editor. The editor now says so rather than rendering nothing.

Verified against PX4 SITL, a local flight-review, and on hardware — a Jetson running this branch against an ARK FMU v6X on PX4 1.18, upgrading a real pre-FTP install: first launch queues only the newest log, a log closing mid-run is picked up automatically (logger cycles via nsh, end to end to a plot link in the onboard Flight Review), select-all fetches and publishes the rest, a restart re-downloads and re-uploads nothing, delete/re-fetch and cancel behave through the gateway, the SSE stream flows unbuffered through nginx and the dev server, and the pre-FTP database plus old-format config carries its state and settings across.

## Summary

Adds a Logs page that lists every log logloader knows about -- on the vehicle,
downloaded, uploaded -- and lets the operator pick which ones to fetch and
publish. Uploaded logs link straight to the plot in the onboard Flight Review.

## Problem

logloader used to decide on its own what to download and upload, and there was
no way to see what it had or to ask for a particular log. Its counterpart
change makes fetching request-driven, which needs somewhere to make requests
from.

## Solution

logloader now serves a small localhost API on :3005; the gateway proxies it at
/api/logloader, including the SSE stream that drives live transfer progress.
The page follows the existing house patterns: Options API, an axios service
module, an inline EventSource with named events, and the SystemPage
loading/error/retry triad.

Select-all or a subset, then Download & Upload, Download Only, Upload, or
Cancel. A log that is still being written by the vehicle never appears, so it
cannot be selected at a size it will not keep.

Verified against PX4 SITL and a local flight-review: first-launch queues only
the newest log, select-all fetches and publishes the remaining four, and each
lands byte-identical with a working plot link.
…ructure

## Summary

Adds a one-time translation of logloader.toml from its flat keys to the nested
layout, run from postinst before merge_configs.py.

## Problem

merge_configs takes its structure from the new template and prunes scalars the
template does not have. That is the right rule for a removed field and the wrong
one for a renamed field. Simulating an upgrade of a config with

    remote_server = "https://review.arkelectron.com"
    email = "pilot@example.com"
    upload_enabled = true
    public_logs = true

produced a merged file with [upload.remote] enabled = false and the stock
review.px4.io url: a vehicle configured to upload to a remote Flight Review
would silently stop after the upgrade. logloader still reads the flat keys as a
fallback, but that does not help when the merge has already deleted them.

## Solution

migrate_logloader_config.py rewrites the flat keys into their new homes, in both
the live config and the pre-upgrade backup, so what reaches merge_configs is an
ordinary same-shape merge. It is idempotent, leaves an already-nested file
alone, and never fails an upgrade.
…findings

## Summary

Follow-up to the Logs page covering an integration review, plus one bug that
turns out to affect every SSE consumer in the app, not just this page.

## Problem

The dev server gzips responses by default and its compression middleware
buffers until it has enough bytes, which for an SSE stream is never. Measured
through `npm run serve` with a browser's Accept-Encoding: 0 bytes and an
EventSource stuck at readyState 0; the same request through the gateway
returned immediately. That silently breaks the journal viewer, the network
stats stream and the firmware progress stream in dev as well.

Separately, http-proxy-middleware v3 reads handlers from `on`, so the v2
`onError`/`logLevel` options every proxy block used were dead: a stopped
upstream answered with the default plain-text 504 rather than the JSON the UI
parses, and the page reported "Request failed with status code 504".

## Solution

`compress: false` on the dev server, and `on: { error }` for all five proxies.

On the page: an upload records only the relative path Flight Review redirects
to, so targets now carry their base url and a remote plot link resolves against
the right origin instead of the ARK-OS host. A dropped stream is reported
rather than leaving a frozen table under a green "connected" dot; confirmations
expire; deleting a downloaded log asks first; an upload request names only the
targets that actually need the log; the select-all box is disabled on an empty
table; and the bokeh websocket is proxied in dev so a plot link works there too.

logloader's config tables are one level deep now, because the shared TomlEditor
renders exactly one level — with two, every upload setting silently vanished
from the Services page config editor. The editor also says so now instead of
rendering nothing, and the upgrade migration writes to the flattened names.
logloader no longer polls the vehicle, so the page asks for a listing when
it loads and offers a Refresh button; the new logging indicator mirrors
MAV_SYS_STATUS_LOGGING so it is visible why transfers wait.
…rades

Supplying on.error in http-proxy-middleware v3 replaces its error plugin,
guards included: an upstream dying mid-SSE made writeHead throw on a
response whose headers were long gone, an uncaught exception in a socket
callback that took the whole gateway down. The guards live in our handler
now, shared by all five proxies.

EventSource only retries network failures; the gateway's 502 while the
daemon restarts closes it for good, so the page retries that case itself
and resyncs what the dead stream missed.

merge_configs prunes scalars the template lacks, and [download] use_burst
was not in it -- an operator's ftp_use_burst=false survived the rename
migration only to be deleted by the merge that follows it.
Points at logloader main after ARK-Electronics/logloader#32: MAVLink FTP
transport, request-driven fetching, event-driven indexing via
MAV_SYS_STATUS_LOGGING, the HTTP API the Logs page drives, and the
review fixes.
@dakejahl
dakejahl merged commit 08a3520 into main Aug 6, 2026
@dakejahl
dakejahl deleted the feat/logs-page branch August 6, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant