Next-Generation Accounting Portal & PEAR Prototype

Status: The accounting portal is in production and live at accounting.egi.eu.

Description

This repository implements the next-generation accounting portal for the Worldwide LHC Computing Grid (WLCG) and European Grid Infrastructure (EGI). It also serves as a prototype and reference implementation for PEAR (Portal for Enhanced Accounting Records), the modernized central accounting system.

It provides unified tracking of heterogeneous computing resources. On the backend, a simplified, schema-flexible ingestion pipeline makes it easy to handle diverse resource types. On the frontend, Grafana dashboards offer an intuitive interface to the data.

Data Flow

Accounting data flows through four stages:

flowchart TB
    APEL[APEL] --> AMS[ARGO AMS]
    AMS --> C

    subgraph pear [" "]
        C["apel-consumer<br/><b>1 · Consume</b>"]
        Q[(Directory Queue)]
        P["apel-parser<br/><b>2 · Parse & Enrich</b>"]
        D["grafana-dashboards<br/><b>4 · Visualize</b>"]
        C --> Q --> P
    end

    CRIC[WLCG CRIC] -- Site Topology --> P
    P <-- "↓ Store (Publish via STOMP)<br/>↑ Read (Request via API)" --> MONIT["CERN MONIT<br/><b>3 · Store</b>"]
    MONIT <--> IDB[(InfluxDB)]
    MONIT --> D

    classDef repo fill:#dbeafe,stroke:#2563eb,color:#0b1f44
    class C,Q,P,D repo

Loading

The highlighted boxes indicate the components of the data flow provided by this repository.

1. Consume: apel-consumer uses SSM to consume APEL accounting records as messages from ARGO AMS and write them to a directory-based queue. It runs as a daemon that continuously waits for new messages.
2. Parse & Enrich: apel-parser runs as a cron job, scheduled to match APEL's publishing cadence. It reads the consumed messages from the file system, splits them into individual records, enriches them with site topology from WLCG CRIC, and aggregates them per month.
3. Store: The resulting monthly summaries are stored in InfluxDB. The parser does not write to the database directly; it publishes the summaries over STOMP to CERN MONIT, which ingests them into InfluxDB.
4. Visualize: grafana-dashboards defines Grafana dashboards for visualizing the data from InfluxDB.

Data for each infrastructure type goes through its own instance of this flow; see One Pipeline per Infrastructure Type.

External Systems

The systems in the diagram that are not highlighted are:

• APEL is the legacy central accounting system for WLCG and EGI. Sites send their accounting data to it via message queues. APEL's central service aggregates the data and publishes summaries via ARGO AMS.
• ARGO AMS is the messaging service used by EGI. APEL messages are published to and consumed from it with SSM, APEL's custom messaging client.
• WLCG CRIC (Computing Resource Information Catalogue) is the WLCG catalogue of computing resources. It is used to add site topology to the accounting data.
• CERN MONIT is CERN's monitoring infrastructure and manages the InfluxDB database. Data is stored by publishing messages to MONIT and read by sending requests to its API.

Repository Layout

apel_consumer/                 1 · Consume
├── Dockerfile                 container image (SSM on AlmaLinux 9)
├── run.sh                     entrypoint: renders the SSM config and starts the receiver
├── receiver.template.cfg      SSM receiver config template
├── dn_allowlist               allowed sender DNs (not in git, needed for the build)
└── k8s/                       grid + cloud Deployments, shared directory queue volume

apel_parser/                   2 · Parse & Enrich, Publish for 3 · Store
├── Dockerfile                 container image (Python)
├── requirements.txt           Python dependencies
├── message_parser.py          entrypoint: parsing, enrichment, aggregation, publishing/storing
├── tools.py                   STOMP publisher, CRIC and MONIT clients
├── constants.py               data model, configuration, topology overrides
└── k8s/                       grid + cloud CronJobs, CVMFS volume

grafana-dashboards/            4 · Visualize
└── portal-overview.json       portal dashboard

Data Model

There is one point per month and tag combination. The timestamp is the first day of the month.

Kind	Names
Tags	vo, tier, country, federation, site, infra, benchmark, ngi
Fields (common)	raw_wc_time, raw_wc_work, raw_cpu_time, raw_cpu_work, raw_cpu_eff
Fields (Grid-only)	job_count
Fields (Cloud-only)	memory_used, disk_used, vm_count, cpu_core_count, network_in, network_out

The site topology tags (tier, country, federation, ngi) come from WLCG CRIC. tier and federation are only resolved for the LHC VOs; other VOs get NON-WLCG-Tier and NON-WLCG-Federation. LHC sites without a federation in CRIC get NON-MOU-Federation. Values that cannot be determined, e.g. for a site missing from CRIC, are UNKNOWN.

The source of truth for the data model is apel_parser/constants.py.

Dashboard

grafana-dashboards/portal-overview.json defines the portal's main Grafana dashboard, which provides an overview of the InfluxDB measurement described in Data Model.

Variables

The dashboard variables at the top control which data the panels show and how it is broken down:

• Filters: every tag of the measurement has a filter, which restricts all panels to the data with the selected tag values. For example, selecting ATLAS in the "VO" filter shows only ATLAS accounting data. Each filter accepts multiple values or "All", and combining filters narrows the data further. The available values are the ones stored in InfluxDB for the selected time range.
  • The "VO" and "Site" filters only list values that match the current selection of the site topology filters. For example, selecting a country limits the "Site" list to the sites in that country.
• Group By: selects the tag by which the data is aggregated (default: site). For example, grouping by vo sums the metrics per VO, so each panel shows one entry per VO.

Panels

The panels are organized into three rows, each showing the same metrics in a different way:

Row	Panel Type	Shows
Historical Charts	Time Series	Evolution over time
Summary	Tables	Totals over a selected time range
Part-to-Whole Charts	Pie Charts	Percentages of the totals

Each row is organized into three subsections, each showing a different set of accounting metrics:

• Common: Grid and Cloud
• High Throughput: Grid-only
• Cloud Compute: Cloud-only

Key Concepts

Stateless Processing of Full History

APEL republishes the complete accounting history every time it publishes, and apel-consumer receives all of it. As a result, apel-parser does not need to keep any state between runs: each run rebuilds the monthly totals from scratch from the messages it receives. The only decision the parser makes is how far back in history to go: it keeps only the data from the last --months months, including the current month, and drops all older records. For example, --months 15 keeps the data from the current month and the 14 months before it.

The one exception is the site topology: the parser reuses the topology of already stored data, as described in Site Topology Fixed on First Store.

Updates by Overwriting

Newly stored data overwrites the existing data. InfluxDB treats points with the same measurement, tags and timestamp as the same point. Since every run has the complete records for each month, the latest accounting data is treated as the correct one.

As a result:

• Corrections to the accounting metrics and late records in APEL reach the dashboard with the next run, without any special handling.
• Stored points are only replaced, never removed. If a correction in APEL changes a tag value, e.g. a record's VO or benchmark, or withdraws a record, the previously stored point is not overwritten: it remains in InfluxDB, is still counted, and must be deleted manually.
• Months older than the selected history window are no longer updated and keep their last stored values.

This relies on each run finding exactly one complete APEL publication in the queue. APEL publications are not timestamped, so the parser cannot tell them apart: two publications in the queue are summed together, leading to double counting, while a publication that is still arriving produces partial totals. This is why the parser's schedule must match APEL's.

Site Topology Fixed on First Store

Overwriting only works if a month is stored again with exactly the same tags. The topology tags, however, come from WLCG CRIC and can change over time, e.g. when the tier of a site changes. If the parser applied the new topology to an already stored month, InfluxDB would see a new series instead of an update, and that month would be counted twice: once under the old tier and once under the new one.

The parser therefore distinguishes two kinds of tags:

• Identity tags (INFLUXDB_SERIES_IDENTITY_TAGS in constants.py) uniquely identify an entry. They never change for the same data.
• Topology tags (INFLUXDB_SITE_TOPOLOGY_TAGS in constants.py) describe a site's topology. They can change.

Before storing new data, the parser reads the points already stored for the same months. For every point to be stored, if exactly one stored point has the same month and identity tags, the stored topology is kept. Otherwise, the current topology from WLCG CRIC is used.

The topology is therefore fixed per series and month: every series keeps, for each month, the topology with which it was first stored. A topology change in WLCG CRIC applies to months that have not been stored yet, i.e. starting with the following month, and to series that appear for the first time in an already stored month, e.g. a VO that starts running at a site late in the month.

One Pipeline per Infrastructure Type

Each infrastructure type has its own pipeline from end to end:

• APEL publishes each type to its own ARGO AMS destination.
• A separate apel-consumer Deployment receives each type in its own directory queue.
• A separate apel-parser CronJob processes each queue, with --infra-type selecting how the records are read.

Records for different infrastructure types have different formats and metrics, and APEL publishes them on different schedules. Keeping them apart allows each pipeline to be scheduled, debugged and re-run on its own, and a failure in one does not affect the other. All pipelines share the same code and store data in the same InfluxDB measurement, where the infra tag tells them apart: Cloud for the cloud pipeline, and Grid or Grid-Local for the grid pipeline, depending on the Infrastructure field of each record. Supporting a new type means adding a new pipeline, plus the parsing logic for its record format.

Deployment

The manifests in apel_consumer/k8s/ and apel_parser/k8s/ describe the Kubernetes deployment at CERN; Deploying Outside CERN lists what needs to be adapted for a deployment elsewhere. Kubernetes is not a requirement: each pipeline is a daemon (ssmreceive) and a cron job (message_parser.py) sharing a queue directory, so it can also run on VMs with the same configuration as in the manifests.

Prerequisites

• A Kubernetes cluster (or VMs) with a ReadWriteMany storage class (or shared file system) for the queue, and CVMFS (or another source of the IGTF CA certificates) for the IGTF trust bundle.
• An ARGO AMS token for the accounting project, with one destination per infrastructure type.
• An SSM certificate and private key, and the DNs of the APEL servers allowed to send messages.
• CERN MONIT credentials for the STOMP broker (storing) and the data source API (reading).
• Grafana with an InfluxDB data source for the database.

1. Build the Images

Create apel_consumer/dn_allowlist (not in git) with one allowed sender DN per line, then build and push the images:

docker build -t registry.cern.ch/wlcg-ops/apel-consumer:latest apel_consumer/
docker build -t registry.cern.ch/wlcg-ops/apel-parser:latest apel_parser/
docker push registry.cern.ch/wlcg-ops/apel-consumer:latest
docker push registry.cern.ch/wlcg-ops/apel-parser:latest

2. Create the Secrets

# apel-consumer: ARGO AMS token, SSM certificate and key
kubectl create secret generic ssm-token --from-literal=token=<AMS token>
kubectl create secret generic globus-certs \
  --from-file=WLCG_usercert.pem=<cert> --from-file=WLCG_userkey.pem=<key>
# apel-parser: MONIT STOMP broker (storing) and data source API (reading)
kubectl create secret generic mq-creds \
  --from-literal=host=<host> --from-literal=port=<port> \
  --from-literal=username=<user> --from-literal=password=<password>
kubectl create secret generic monit-datasources-creds \
  --from-literal=api=<URL> --from-literal=token=<token>

3. Apply the Manifests

Apply the volumes first, then the consumers, then the parsers:

kubectl apply -f apel_consumer/k8s/apel-consumer-output-pvc.yaml -f apel_parser/k8s/cvmfs-pvc.yaml
kubectl apply -f apel_consumer/k8s/apel-consumer-grid-deployment.yaml -f apel_consumer/k8s/apel-consumer-cloud-deployment.yaml
kubectl apply -f apel_parser/k8s/apel-parser-grid-cronjob.yaml -f apel_parser/k8s/apel-parser-cloud-cronjob.yaml

The consumers start waiting for messages right away, and the parsers run on the schedules in their CronJobs.

4. Import the Dashboard

Import grafana-dashboards/portal-overview.json into Grafana, selecting the InfluxDB data source and setting the measurement name.

Deploying Outside CERN

The manifests and images contain a few CERN-specific values. For another deployment, change them as follows:

• Images: push the images to your own registry and update the image: fields in all four manifests. In apel_consumer/Dockerfile, the CERN AlmaLinux 9 base image can be replaced by any AlmaLinux 9 image (e.g. almalinux:9).
• Queue volume: in apel_consumer/k8s/apel-consumer-output-pvc.yaml, replace the meyrin-cephfs storage class with a ReadWriteMany storage class available in your cluster (e.g. CephFS or NFS).
• Trust bundle: the parser needs the IGTF CA certificates to connect to WLCG CRIC. At CERN they come from CVMFS, via the cvmfs PVC and IGTF_TRUST_BUNDLE_PATH in the CronJobs. Either provide CVMFS in your cluster, or install the certificates another way (e.g. the ca-policy-egi-core package) and point IGTF_TRUST_BUNDLE_PATH to them, removing the cvmfs volume.
• ARGO AMS: in apel_consumer/receiver.template.cfg, change the cern- prefix of destination to match the destinations assigned to you. Put your AMS token in the ssm-token secret, and the DNs of the APEL servers sending to you in dn_allowlist.
• MONIT: storing and reading go through MONIT (STOMP broker, data source API, InfluxDB database and measurement). If your setup also accepts documents over STOMP and exposes the InfluxDB 1.x query API (InfluxQL over HTTP, with token authentication), adjusting the secrets and environment variables is enough. Otherwise, replace publish and fetch_historical_topology in apel_parser/tools.py, and the MONIT-specific metadata fields (idb_tags, producer, type) that build_docs in apel_parser/message_parser.py adds to each document.
• Grafana: when importing the dashboard, select your InfluxDB data source and set the measurement name.

Configuration

The components are configured through the environment variables and command-line options in the manifests. The parser's environment variables are defined in apel_parser/constants.py. Its command-line options are:

Option	Purpose
-o, --output-dir	Directory for the output files (see Output Files)
--messages-dir	Directory-based queue from which to read the messages
--infra-type	Infrastructure type of the messages (grid or cloud)
-m, --months	How many months of history to keep, including the current month (default: 1, i.e. only the current month)
--lhc-only	Keep only the records of LHC VOs
-p, --publish	Publish the results; without it, the parser only writes the output files

Output Files

Each parser run also writes its results as JSON files to <output-dir>/<UTC timestamp>/:

• {grid,cloud}_accounting_data_<Y>_<M>.json: the monthly summaries that get stored.
• grid_accounting_data_ce_<Y>_<M>.json (grid only): the same data broken down per CE (SubmitHost). It is written locally but not stored in InfluxDB.

The parser never deletes these files, so they accumulate on the shared volume unless another process cleans them up.

Running the Parser Locally

The parser requires Python 3.11 or later.

pip install -r apel_parser/requirements.txt
python apel_parser/message_parser.py \
  -o ./parser_output \
  --messages-dir /path/to/queue/grid/incoming \
  --infra-type grid \
  -m 1

Even without --publish, the parser reads from WLCG CRIC and MONIT, so it needs the same environment variables as the CronJob, except for the MESSAGE_* and MQ_* ones. IGTF_TRUST_BUNDLE_PATH can point to a local CVMFS mount or any CA bundle that trusts WLCG CRIC. The parser also removes the messages it processes from the directory queue, so work on a copy.

Operations

All files a deployment writes are on the shared volume at /var/spool/apel: the queues in <grid,cloud>/incoming/, the consumer logs in ssmreceive-<grid,cloud>.log, and the parser's output files in parser_output/. The parser itself logs to standard output, so its logs are available through kubectl logs on the CronJob's pods.

• Messages rejected by SSM, e.g. from a sender DN not in the allowlist, end up in <grid,cloud>/reject/.
• A parser run that finds an empty queue does nothing. This is expected when APEL has not published since the last run.
• Sites missing from WLCG CRIC are logged, and their data is stored with UNKNOWN topology tags.