Improve memory configuration documentation and validation for InfluxDB3 Enterprise

[dburton-influxdata]

---

title: "Improve memory configuration documentation and validation for InfluxDB3 Enterprise"
labels:

• documentation
• enhancement
• influxdb3-enterprise

---

Summary

Add comprehensive memory configuration documentation and validation to prevent common OOM (Out of Memory) kills in InfluxDB 3 Enterprise deployments, particularly affecting the compactor component.

Problem

Customer deployments are experiencing OOM kills due to memory overcommitment from misconfigured cache settings. Current issues:

1. No memory configuration guidance in HELM chart documentation
2. No validation of memory settings during chart installation
3. Percentage-based cache configuration is ambiguous (INFLUXDB3_PT_FILE_CACHE_MAX_BYTES: 50%)
4. Conflicting defaults that lead to cache allocation > container limit
5. No warnings when force snapshot threshold is misconfigured

Real-World Example (VAST Customer)

Customer configured:

resources:
  limits:
    memory: 95Gi

env:
  - name: INFLUXDB3_PT_FILE_CACHE_MAX_BYTES
    value: 50%                              # 47.5GB
  - name: INFLUXDB3_OBJECT_STORE_CACHE_MAX_SIZE_BYTES
    value: "34359738368"                    # 32GB
  - name: INFLUXDB3_PT_COMPACTOR_INPUT_SIZE_BUDGET
    value: 12GB

Result:

• Total cache allocation: 91.5GB (96% of limit)
• Available for operations: 3.5GB (4% of limit)
• Compactor OOMKilled after 38 minutes

Root cause: Caches alone consumed 96% of available memory, leaving insufficient space for compaction operations.

Proposed Solution

1. Add Memory Sizing Documentation

Create comprehensive memory sizing guide covering:

• Memory budget calculation
• Component-by-component sizing guidelines
• Pre-configured profiles (small/standard/large/high-performance)
• Troubleshooting guide
• Validation checklist

Documentation structure:

docs/
  memory-sizing-guide.md          # Comprehensive guide with calculator
  memory-troubleshooting.md       # OOM troubleshooting
  examples/
    memory-config-64gb.yaml       # Small deployment
    memory-config-95gb.yaml       # Standard deployment
    memory-config-150gb.yaml      # Large deployment
    memory-config-200gb.yaml      # High-performance deployment

2. Add HELM Chart Validation

Add values.schema.json to validate memory configuration at install/upgrade time.

Validation checks:

1. Warn if total cache allocation > 80% of container memory
2. Error if force snapshot threshold > 60% of container memory
3. Warn if force snapshot threshold > total cache allocation
4. Warn if percentage-based cache configuration used in production
5. Recommend absolute byte values instead of percentages

Example validation output:

⚠️  WARNING: Memory overcommitment detected
    Total cache allocation: 91.5GB (96% of 95GB limit)
    Recommended maximum: 76GB (80% of limit)
    
    Current configuration:
      File cache:              47.5GB (50%)
      Object store cache:      32.0GB
      Compactor input budget:  12.0GB
      
    Recommendation:
      Use one of the pre-configured memory profiles:
        helm install ... -f examples/memory-config-95gb.yaml
      
    Or reduce cache sizes manually. See:
      docs/memory-sizing-guide.md

3. Update Default Values

Change values.yaml defaults to be safer:

Current (dangerous):

compactor:
  env:
    - name: INFLUXDB3_PT_FILE_CACHE_MAX_BYTES
      value: 50%     # Ambiguous, error-prone
    - name: INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD
      value: 70%     # Too high

Proposed (safe):

compactor:
  env:
    # Use absolute values by default (for 95GB container)
    - name: INFLUXDB3_PT_FILE_CACHE_MAX_BYTES
      value: "21474836480"     # 20GB (21% of 95GB)
    
    - name: INFLUXDB3_OBJECT_STORE_CACHE_MAX_SIZE_BYTES
      value: "21474836480"     # 20GB (21% of 95GB)
    
    - name: INFLUXDB3_PT_COMPACTOR_INPUT_SIZE_BUDGET
      value: "8GB"
    
    - name: INFLUXDB3_EXEC_MEM_POOL_BYTES
      value: "21474836480"     # 20GB (21% of 95GB)
    
    - name: INFLUXDB3_DATAFUSION_NUM_THREADS
      value: "8"     # Limit parallelism
    
    - name: INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD
      value: "50%"   # Reduced from 70%

Memory budget with safe defaults:

File cache:              20 GB (21%)
Object store cache:      20 GB (21%)
Compactor input budget:   8 GB (8%)
Exec mem pool:           20 GB (21%)
Runtime overhead:         5 GB (5%)
                        ---------
Total allocated:         73 GB (77%)
Container limit:         95 GB
Available for ops:       22 GB (23%) ✅ SAFE

4. Add Pre-Configured Memory Profiles

Add example configurations for common deployment sizes:

examples/memory-config-64gb.yaml (Small deployment)

• Container: 64GB
• File cache: 15GB
• Object cache: 10GB
• Input budget: 5GB
• Exec pool: 10GB
• Total allocated: 43GB (67%)

examples/memory-config-95gb.yaml (Standard deployment)

• Container: 95GB
• File cache: 20GB
• Object cache: 20GB
• Input budget: 8GB
• Exec pool: 20GB
• Total allocated: 73GB (77%)

examples/memory-config-150gb.yaml (Large deployment)

• Container: 150GB
• File cache: 40GB
• Object cache: 30GB
• Input budget: 15GB
• Exec pool: 30GB
• Total allocated: 123GB (82%)

examples/memory-config-200gb.yaml (High-performance deployment)

• Container: 200GB
• File cache: 50GB
• Object cache: 40GB
• Input budget: 20GB
• Exec pool: 40GB
• Total allocated: 160GB (80%)

Usage:

helm install influxdb3-enterprise influxdata/influxdb3-enterprise \
  -f examples/memory-config-95gb.yaml \
  -f my-custom-values.yaml

5. Add Memory Monitoring Dashboard

Add Grafana dashboard with memory metrics:

• Container memory usage (%)
• Cache utilization (file, object)
• Execution pool utilization
• Forced snapshot rate
• OOMKill event count

Alert thresholds:

• ⚠️ Warning at 80% memory usage
• 🔥 Critical at 90% memory usage
• 🔥 Critical on any OOMKill event

Implementation Checklist

Phase 1: Documentation (High Priority)

• Create docs/memory-sizing-guide.md
• Create docs/memory-troubleshooting.md
• Update main README with memory sizing section
• Add memory configuration to troubleshooting guide

Phase 2: Example Configurations (High Priority)

• Create examples/memory-config-64gb.yaml
• Create examples/memory-config-95gb.yaml
• Create examples/memory-config-150gb.yaml
• Create examples/memory-config-200gb.yaml

Phase 3: Chart Validation (Medium Priority)

• Add values.schema.json with memory validation rules
• Implement validation logic in _helpers.tpl
• Add memory budget calculator helper function
• Display warnings during helm install/upgrade

Phase 4: Update Defaults (Medium Priority)

• Change default cache values to absolute bytes
• Reduce force snapshot threshold from 70% to 50%
• Add INFLUXDB3_EXEC_MEM_POOL_BYTES to defaults
• Add INFLUXDB3_DATAFUSION_NUM_THREADS to defaults

Phase 5: Monitoring (Low Priority)

• Create Grafana dashboard JSON
• Add Prometheus recording rules for memory metrics
• Add alerting rules for OOMKill events
• Document dashboard installation

Testing Plan

1. Validation testing:

   • Test that validation catches overcommitted memory
   • Test that validation warns about percentage-based configs
   • Test that validation accepts safe configurations

2. Configuration testing:

   • Deploy with each pre-configured profile
   • Run compaction workload for 2+ hours
   • Verify no OOM kills
   • Verify memory stays under 85% of limit

3. Upgrade testing:

   • Upgrade existing deployment with new defaults
   • Verify no disruption to running workload
   • Verify memory usage improves

Benefits

1. Prevents OOM kills by providing safe default configurations
2. Educates users with comprehensive documentation
3. Validates early to catch misconfigurations before deployment
4. Saves time with pre-configured profiles for common scenarios
5. Improves observability with memory monitoring dashboard

Related Issues

• influxdb3-enterprise: Startup probe timeout too short for v3.2+ #800 - Initial VAST customer issue (startup probe timeout)
• Fix: Increase startup probe timeout for influxdb3-enterprise 3.2+ #801 - PR fixing startup probe timeout
• Comment: Fix: Increase startup probe timeout for influxdb3-enterprise 3.2+ #801 (comment)

Additional Context

Full analysis and documentation already created:

• COMPACTOR-OOM-ANALYSIS.md - Deep technical analysis (23KB)
• COMPACTOR-OOM-QUICK-FIX.md - Customer quick fix guide (19KB)
• MEMORY-SIZING-GUIDE.md - Comprehensive sizing guide (18KB)

These documents are available and can be adapted for inclusion in the HELM chart repository.

[tomklapka]

While implementing the validation proposed here I went looking for the failure this would have caught, and the evidence points at a different cause. Sharing it before anyone builds on the current framing.

InfluxDB 3.11 added --compacted-data-skip-file-index, whose description reads: "The file index maps indexed column values to parquet files (…) On deployments where the index has grown near host memory it can make nodes unable to start; skipping it trades query-time file pruning (queries stay correct, time-range pruning still applies) for a small, bounded memory footprint at startup." That is a close match for the compactor OOM reported here, and for the support cases where nodes stay unready for tens of minutes.

Three things argue against cache over-allocation being the cause. An earlier case with the same signature was resolved by changing startup behaviour rather than cache configuration. In one reported case the affected pod was using a small fraction of its memory limit while failing to become ready, so cache sizing was not the constraint. And both of the options that bound this work are new in 3.11, which fits them being added in response to exactly these reports.

There is also a practical problem with the validation itself: compactor-input-size-budget defaults to 50% of detected RAM, and several other options have similar server-side defaults. A chart-side budget check that only sums explicitly set values would pass a configuration that is over-allocated in practice, and to avoid that the chart would have to mirror the product's defaults and keep them in sync across versions.

I have opened #828 exposing compactedData.skipFileIndex and compactedData.loadConcurrencyLimit instead, which lets operators bound the startup load directly. Worth noting for anyone hitting this on 3.9 or 3.10: neither option exists there, so an upgrade to 3.11 comes first.

Suggest we refocus this issue on startup memory, or close it in favour of #828. The sizing documentation and example profiles it proposes are still worth having, they are just a separate piece of work from the OOM.

[tomklapka]

Correction to my previous comment: I was wrong that these options do not exist before 3.11.

Both shipped earlier under the enterprise-prefixed names, INFLUXDB3_ENTERPRISE_COMPACTED_DATA_SKIP_FILE_INDEX and INFLUXDB3_ENTERPRISE_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT, and are present on the 3.9 and 3.10 branches. 3.11 dropped the ENTERPRISE_ prefix and keeps the old names working as aliases. I had only searched for the un-prefixed spelling.

This matters for anyone acting on the earlier comment: an upgrade to 3.11 is not a prerequisite. A cluster on 3.9 or 3.10 can set the enterprise-prefixed variable today, through extraEnv, without waiting for a version upgrade or for a chart release.

The rest of the comment stands: the evidence still points at the compaction file index rather than cache over-allocation. #828 has been updated to emit both spellings so the chart works on either.

[tomklapka]

Chart 0.11.0 is out and carries the first half of this. The two compacted-data options are now first-class values rather than something to spell out in extraEnv:

compactedData:
  skipFileIndex: true
  loadConcurrencyLimit: 8

The ConfigMap emits both the 3.11 names and the INFLUXDB3_ENTERPRISE_ ones, so a release works against 3.9 and 3.10 images too and the upgrade caveat from my earlier comment still does not apply. The README has a Compacted-Data Startup section and TROUBLESHOOTING covers the symptom.

One caveat worth stating plainly: these are Parquet-engine only. Once a node has fully adopted PachaTree the machinery that reads them is never built and the server logs that they can be removed from the configuration. They stay live throughout the migration itself, so they help exactly where this issue bites and are redundant on a cluster that has finished the move.

What 0.11.0 does not deliver is the memory budget part of this issue: validation that cache settings do not add up past the container limit, and the guidance around the example above. Nothing in the chart checks that today, so this is worth keeping open for it.