Skip to content

operations-debugging.md

Latest commit

 

History

History
504 lines (363 loc) · 25.6 KB

operations-debugging.md

File metadata and controls

504 lines (363 loc) · 25.6 KB
Raw

🛠️ Operations & Debugging (CLI & DB Dumps)

Tuliprox is designed as a "Fire & Forget" stream broker. However, when streams stutter, provider connection limits block your users, or EPG data and TMDB covers do not match, the engine provides deep, low-level insights under the hood.

This chapter covers the Command Line Interface (CLI), Logging architecture, and the internal Database Viewers.

1. Command Line Arguments (CLI Flags)

While Tuliprox is usually run via Docker, understanding the CLI flags is crucial for debugging and manual interventions.

Flag Purpose & Technical Background
-s, --server Starts continuous Server Mode (API, Web UI, Background Workers). Without this flag, Tuliprox acts as a "One-Shot" playlist generator that downloads, processes, and immediately exits.
-H, --home <DIR> Sets the Home Directory. All relative paths in the configuration are resolved against this directory. If not set, resolves via TULIPROX_HOME env variable, or finally the binary's directory.
-c, -i, -a, -m, -T Overrides specific config paths (e.g., -c /etc/tuliprox/config.yml). Useful for testing experimental configurations without altering the production setup.
-t, --target <NAME> Targeted Processing: Forces processing of the specified target only. Crucial: This bypasses the enabled: false state in the config! Extremely useful to quickly re-render a broken list via cron/shell without blocking the entire system with other heavy targets.
--genpwd Interactively generates a secure Argon2id password hash for the user.txt file. Never store plaintext passwords!
--healthcheck Docker Support: Pings the API over localhost. Returns Exit Code 0 if the server responds with {"status": "ok"}.
--sh <QUERY> Stream History Viewer: Dumps and filters stream history records from binary archive files. Accepts inline JSON or @file.json. See Stream History Viewer below.
--scan-library Triggers an incremental scan of the local media directory (if configured).
--force-library-rescan Ignores modification timestamps and forces a full TMDB/PTT re-evaluation of all local media files.
--inspect-db <PATH> Database Inspection: Inspects a B+Tree database file. Reports file size, storage version (V2/V3), detected type, healthy entries, and count of corrupted records. See Database Migration & Inspection below.
--migrate-db <PATH> Database Migration: Migrates a legacy (V1/V2) database to B+Tree V3. Automatically creates a timestamped backup before migrating. Recovers all healthy records and bypasses damaged entries.
--db-type <TYPE> Explicit database type override for inspection/migration (series, xtream, target-xtream, m3u, target-m3u, epg, etc.). Auto-detected if omitted.
--no-backup Skips creating a .bak.<timestamp> file before running --migrate-db.

2. Logging Levels and Module Filtering

Tuliprox utilizes the powerful Rust env_logger crate. The log verbosity can be controlled at an extremely granular level via config.yml (log.log_level), the environment variable TULIPROX_LOG, or the CLI flag -l.

The evaluation hierarchy is: CLI Argument > Env-Var > config.yml > Default (info).

Available levels: trace, debug, info, warn, error.

The Magic of Module Filtering: Often, you do not want to set the entire system to trace (which would flood your console and disk), but rather investigate a specific algorithm. You can pass comma-separated module paths:

# Everything on Info, but the internal Mapper on Trace
# (Useful to make print() commands from the DSL visible!):
./tuliprox -s -l "info,tuliprox::foundation::mapper=trace"

# Show me all low-level HTTP-Connection errors from the Hyper crate:
./tuliprox -s -l "info,hyper_util::client::legacy::connect=error"

Note: If log.sanitize_sensitive_info is set to true in the config (default), Tuliprox masks passwords, provider URLs, and external client IPs in the logs with ***. This is strongly recommended so you can safely share logs on GitHub or Discord!

3. Database Dumps (B+Tree Analysis)

Tuliprox is built to be extremely resource-efficient. It does not keep massive playlists (often > 200,000 entries) permanently in RAM. Instead, it stores all parsed metadata, enriched by FFprobe and TMDB, in highly optimized local B+Tree Database files (.db).

Sometimes you need to know exactly what Tuliprox has discovered in the background about a specific stream. Using the built-in dump flags, you can output these binary files in clean JSON format to your console (or pipe them into a file).

You must point the flag directly at the corresponding .db file inside your storage_dir (e.g., /app/data/):

Flag & Example Usage & Purpose
--dbx <PATH>
./tuliprox --dbx ./data/input_name/xtream/video.db		 Xtream DB: Reads the metadata derived from the Xtream API. Shows you the final JSON payloads with resolved TMDB IDs, extracted video codecs (e.g., H264), and bitrates.
--dbm <PATH>
./tuliprox --dbm ./data/input_name/m3u.db		 M3U Playlist DB: Reads the raw M3U entries. Ideal for seeing how the fallback logic for Tvg-ID or Virtual_ID reacted to messy provider tags.
--dbe <PATH>
./tuliprox --dbe ./data/input_name/xtream/epg.db		 EPG DB: Prints the fully matched XMLTV grid. You see a list of all programmes with their correct Unix timestamps.
--dbms <PATH>
./tuliprox --dbms ./data/input_name/metadata_retry_state.db		 Metadata Retry Status (Cooldowns): Extremely important! Shows you the asynchronous backoff state. If TMDB finds no info for a stream, it lands in a cooldown here. Shows attempts: 3, last_error: "404 Not Found", cooldown_until_ts: 1740000000. This explains why a movie isn't being updated.
--dbv <PATH>
./tuliprox --dbv ./data/target_name/id_mapping.db		 Target-ID Mapping: Tracks the stability of stream UUIDs across updates. Shows which original Provider-ID points to which internal Virtual-ID.
--dbq <PATH>
./tuliprox --dbq ./data/qos_snapshot.db		 QoS Snapshot DB: Dumps the aggregated QoS snapshots used as the future failover input. Shows per-stream identity, daily buckets, and the current 24h/7d/30d score/confidence windows.

Example output via --dbms

{
  "Stream_ID_4242": {
    "resolve": {
      "attempts": 3,
      "next_allowed_at_ts": 1718000000,
      "cooldown_until_ts": 1718604800,
      "last_error": "TMDB lookup completed without matching result",
      "tmdb": null,
      "updated_at_ts": 1718604910
    }
  }
}

Diagnosis: This dump immediately tells you: Tuliprox tried three times to find the movie on TMDB, failed every time, and has now paused this movie until cooldown_until_ts (e.g., 7 days in the future) to save API traffic and prevent rate-limiting.

4. Reading QoS Snapshots

The QoS snapshot DB is the condensed operational view built from raw stream history. It is intended to answer questions like:

  • Which stream is currently the most reliable?
  • Is a stream failing before startup, during first byte, or later during playback?
  • Is the problem likely provider instability, capacity pressure, or transient churn?

What a Snapshot Represents

Each snapshot is keyed by a stable stream identity and stores rolling windows:

  • 24h — recent behavior, best for spotting active incidents or current degradations
  • 7d — medium-term stability, useful to smooth out one-off noise
  • 30d — long-term baseline, useful to judge whether a stream is generally trustworthy

Each window contains:

  • score — compact reliability score (0-100)
  • confidence — how much data the score is based on
  • startup counters
  • runtime abort counters
  • provider-close counters
  • reconnect burden
  • latency and session-duration averages

How To Interpret score

The score is a weighted operational quality estimate:

  • High score (80-100): recent connects are successful, disconnects are rare, first-byte failures are low, and reconnect burden is low
  • Medium score (50-79): stream is usable, but shows noticeable instability or intermittent provider issues
  • Low score (0-49): stream is currently risky; repeated startup failures, runtime aborts, or provider-side churn are dominating

The exact weighting is intentionally pragmatic, not academic. It is designed to provide a stable operational signal for later failover ranking, not a mathematically "perfect" SLA model.

How To Interpret confidence

confidence answers: "How much should I trust this score?"

  • High confidence: enough events were seen in the window to make the score meaningful
  • Low confidence: too little recent traffic; the score may be technically correct but statistically weak

Operationally:

  • treat high score + high confidence as a strong candidate
  • treat high score + low confidence as promising but not yet proven
  • treat low score + high confidence as a real reliability warning

Common Diagnosis Patterns

1. Low score, high connect_failed_count

Startup path is unstable. Look at:

  • startup_capacity_failure_count
  • provider_open_failure_count

Interpretation:

  • high capacity failures => provider/user capacity is too tight
  • high provider-open failures => upstream is unstable before streaming even starts

2. Good startup, but high first_byte_failure_count

Tuliprox could open the session, but the provider never became stream-ready in time. This often points to bad upstream responsiveness or unstable pre-stream behavior.

3. Good startup, but high runtime_abort_count / provider_closed_count

The stream starts, then dies later. This is the classic "provider instability during playback" case and is usually more relevant for later failover ordering than pure startup success.

4. High avg_provider_reconnect_count

The stream survives, but only because Tuliprox has to reconnect repeatedly behind the scenes. This is a reliability warning even if users do not immediately see hard failures.

5. High score in 30d, bad score in 24h

Usually indicates a current incident rather than a historically bad stream. Treat as a recent degradation, not a permanently bad source.

6. Strong 24h score, weak 30d score

Usually indicates recent recovery. Good sign, but wait for confidence to grow before treating it as fully stable again.

QoS Snapshots vs Raw Stream History

Use QoS snapshots when you want a compact answer:

  • "Which streams are strong?"
  • "Which providers are degrading?"
  • "Is this a capacity problem or a runtime stability problem?"

Use raw stream history when you need event-level truth:

  • exact disconnect reasons
  • exact timestamps
  • per-session provider metadata
  • startup vs disconnect timeline reconstruction

The intended workflow is:

  1. Look at QoS snapshots first for ranking and trend detection.
  2. Drill into raw stream history when a snapshot looks suspicious or degraded.

Practical Workflow

# Dump the QoS snapshot DB
./tuliprox --dbq ./data/qos_snapshot.db

# Then inspect raw history for one affected day
./tuliprox --sh '{"from":"2026-04-02","filter":{"provider_name":"acme"}}'

Important Limitation

QoS snapshots are intentionally operational summaries, not a complete event archive:

  • they are ideal for ranking and triage
  • they are not a replacement for raw stream history
  • they are also not yet the failover engine itself

The future failover feature is expected to consume these snapshots, but the snapshots already stand on their own as a debugging and reliability-analysis tool.

5. Stream History Viewer

When stream history is enabled (stream_history in reverse_proxy config), Tuliprox persists connect/disconnect records to daily binary files. The --sh CLI flag lets you query and filter these records offline without starting the server.

Query Format

The --sh flag accepts a JSON query, either inline or via @file.json:

# Inline query: all records from a single day
./tuliprox --sh '{"from":"2026-03-22"}'

# Date range with filter
./tuliprox --sh '{"from":"2026-03-20","to":"2026-03-22","filter":{"api_username":"alice"}}'

# Query from file
./tuliprox --sh @query.json

Query Fields

Field Type Description
from string Start date/datetime. Formats: YYYY-MM-DD, YYYY-MM-DD HH:MM, YYYY-MM-DD HH:MM:SS. At least from or to is required.
to string End date/datetime. Same formats as from. Date-only values expand to end of day (23:59:59).
path string Stream history directory. Defaults to data/stream_history.
filter object Key-value filters applied per record. See filter syntax below.

When only from or to is provided, the query expands to the full UTC day. All timestamps are interpreted as UTC.

Filter Syntax

Filters are key-value pairs where the key is a record field name:

Syntax Meaning Example
"field": "value" Exact match (case-insensitive) "api_username": "alice"
"field": "~regex" Regex match (prefix with ~) "provider_name": "~^acme.*"
"session_id": "42" Numeric exact match "session_id": "42"

Filterable fields: event_type, api_username, provider_name, provider_username, item_type, title, group,
country, source_addr, disconnect_reason, session_id.

Output

Output is a streaming JSON array to stdout. Warnings and errors go to stderr.

[
  {
    "schema_version": 1,
    "event_type": "connect",
    "event_ts_utc": 1742601600,
    ...
  },
  {
    "schema_version": 1,
    "event_type": "disconnect",
    "event_ts_utc": 1742605200,
    ...
  }
]

Examples

# All disconnects with provider errors on March 22
./tuliprox --sh '{"from":"2026-03-22","filter":{"event_type":"disconnect","disconnect_reason":"provider_error"}}'

# All activity for user "bob" in a date range, piped to jq
./tuliprox --sh '{"from":"2026-03-20","to":"2026-03-25","filter":{"api_username":"bob"}}' | jq '.[] | {ts: .event_ts_utc, type: .event_type}'

# Query with regex filter for provider names starting with "acme"
./tuliprox --sh '{"from":"2026-03-22","filter":{"provider_name":"~^acme"}}'

6. Database Migration & Inspection (migrate_db.sh & CLI)

Tuliprox uses a custom B+Tree storage engine. Starting with version 3.3.x, the engine uses the V3 storage format (a slotted-page architecture with page-level checksums, write-ahead logging, and crash recovery).

During startup or container updates, Tuliprox automatically migrates legacy V1 and V2 database files (such as series.db, live.db, video.db, m3u_*.db, epg.db) to V3.

Standalone Migration & Inspection Script (bin/migrate_db.sh)

If you want to manually verify, repair, or migrate a database file before starting the server—or if you need to run migration inside a running Docker container—use the helper script in bin/:

# Show usage and available options
./bin/migrate_db.sh --help

Inspecting a Database (Read-Only)

To check storage version, entry counts, and detect corrupted records without modifying anything on disk:

./bin/migrate_db.sh --inspect /path/to/series.db

Output example:

=== B+Tree Database Inspection ===
Path:              /data/input_dir/series.db
File Size:         27800915 bytes
Storage Version:   V2
Database Type:     input_xtream
Healthy Entries:   22347
Corrupted Entries: 0

Migrating a Local Database (with Automatic Backup)

To convert a legacy V2 database to V3:

./bin/migrate_db.sh /path/to/series.db

This creates a safety backup (series.db.bak.YYYYMMDD_HHMMSS), reconstructs all healthy records into V3 format, verifies the tree integrity, and atomically replaces the file.

Output example:

=== B+Tree Database Migration Complete ===
Path:              /data/input_dir/series.db
Version:           V2 -> V3
Database Type:     input_xtream
Backup Created:    /data/input_dir/series.db.bak.20260905_071212
Recovered Entries: 22347
Corrupted Skipped: 0
Database ID:       a84db1e503b942319e0374beb6d0e08b
Generation:        1
Status:            SUCCESS (valid V3 database)

Running in Docker

If Tuliprox runs inside a Docker container (e.g., named tuliprox):

# Inspect inside container:
./bin/migrate_db.sh --docker tuliprox --inspect /app/data/input_dir/series.db

# Migrate inside container:
./bin/migrate_db.sh --docker tuliprox /app/data/input_dir/series.db

Alternatively, invoke tuliprox directly via docker exec:

docker exec -it tuliprox tuliprox --migrate-db /app/data/input_dir/series.db

7. Hot Reloading Caveats

Tuliprox supports hot-reloading for specific files (mapping.yml, api-proxy.yml) if config_hot_reload: true is set in config.yml.

Important Note for Docker Bind Mounts: If you edit a file on your host system that is bind-mounted into the container (e.g., nano /home/user/tuliprox/config/mapping.yml), the file watcher might report the inotify event using the original host path instead of the container's mount point /app/config/mapping.yml. Tuliprox attempts to resolve this, but depending on your host OS (Windows/WSL vs Linux), filesystem events can be flaky. If hot-reload fails to trigger, a container restart (docker restart tuliprox) is the safest fallback.

8. Runtime Liveness Watchdog

A reverse proxy can wedge without crashing: the process is still alive, but the async scheduler stops making progress and the logs simply stop. At the OS level this looks like an idle process, so docker stats and ps cannot tell it apart from a healthy one.

The watchdog detects exactly that. It is opt-in and off by default.

What it does

  • Heartbeat (inside the runtime): a small task records a monotonic timestamp every second and measures how late it was scheduled.
  • Watchdog thread (outside the runtime): a separate std::thread watches the heartbeat age. If the scheduler is wedged, the heartbeat stops advancing and the thread logs a diagnostic snapshot.
  • On a stall it logs (once, then every 30 s while it continues) the heartbeat age, runtime metrics (worker count, alive tasks, global queue depth, per-worker park counts) and, on Linux, a /proc/self/task inventory with each thread's id, name, state and waiting channel.
  • On recovery it logs that the heartbeat is back.

The watchdog itself only reports. In mode 2 it can additionally exit the process so a supervisor restarts it (see below).

Modes

TULIPROX_WATCHDOG is a mode selector, not a boolean:

Value Mode Behaviour
unset, 0, anything else off No heartbeat task, no watchdog thread.
1, true, on, yes, enabled observe Detect and log stalls; never touches the process.
2, restart restart Like observe, and exit the process after the stall persists past the grace period.

In restart mode the process exits with code 75 (non-zero, so both restart: unless-stopped and restart: on-failure bring it back). This is a deliberate crash-and-restart, not an in-process recovery. Only enable it if a supervisor is actually configured; otherwise the container will stay down.

Settings

Variable Default Purpose
TULIPROX_WATCHDOG (unset = off) Mode selector: 1 = observe, 2 = observe and restart on a confirmed stall.
TULIPROX_WATCHDOG_HEARTBEAT_MS 1000 Heartbeat interval.
TULIPROX_WATCHDOG_STALL_MS 10000 Heartbeat age above which a stall is reported.
TULIPROX_WATCHDOG_RELOG_MS 30000 Re-log interval while a stall continues.
TULIPROX_WATCHDOG_RESTART_GRACE_MS 30000 Mode 2 only: how long the stall must persist before the process exits.

Docker example:

services:
  tuliprox:
    restart: unless-stopped
    environment:
      # observe only
      - TULIPROX_WATCHDOG=1
      # ...or observe and restart on a confirmed stall (needs restart policy above)
      # - TULIPROX_WATCHDOG=2

Healthcheck

While the watchdog runs, GET /healthcheck gains a runtime object:

{
  "status": "ok",
  "runtime": {
    "status": "alive",
    "heartbeat_age_ms": 12,
    "heartbeat_interval_ms": 1000,
    "stall_threshold_ms": 10000,
    "ticks": 8401,
    "stall_episodes": 0,
    "max_schedule_delay_ms": 3,
    "uptime_ms": 8401000,
    "restart_on_stall": false
  }
}

restart_on_stall is true only in mode 2, so an operator can see whether the process is armed to restart itself.

runtime.status is stalled once the heartbeat stops. The top-level status intentionally stays ok for as long as the HTTP server answers, so no orchestrator restarts the process. If the whole runtime is wedged, the endpoint cannot answer at all — in that case the watchdog log is the signal.