Skip to content
corifeoPublic

About

Fast, boring ETL for public vulnerability data: NVD, CISA KEV, EPSS, CWE, CAPEC and Microsoft Patch Tuesday in one columnar dataset for pandas, polars, Parquet or your warehouse.

Topics

Resources

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

vulnflow

Fast, boring ETL for public vulnerability data. NVD, CISA KEV, EPSS, CWE, CAPEC and Microsoft's Patch Tuesday feed, normalized into one columnar dataset for pandas, polars or Parquet.

Python 3.11+ License: MIT Typed CI Data: NVD · KEV · EPSS · CWE · CAPEC · MSRC

Install · Quick start · Full library · Real-data KEV timing · Log4Shell case study · The 2026 surge · What the dates mean · Pipelines · Sources

The quick start on NVD's Recent feed: the stages that ran and how complete each field came out

If you've spent half a notebook cleaning CVE feeds before doing any analysis, vulnflow is for you. It moves public vulnerability data and makes it sane:

  • One line to a dataframe. CacheBuilder(recent_only=True).build().to_pandas() pulls the last week of NVD with KEV, EPSS, CWE and flattened CVSS in seconds. All of NVD takes about two minutes.
  • Columnar all the way. Data lives in NumPy arrays, not record objects. CVD event inference and state for 83,000 CVEs takes under three seconds.
  • Explicit pipelines. Extractors, transforms and loaders run exactly as you list them, in dependency order. A missing dependency is reported, not silently skipped.
  • Typed and tested. py.typed, mypy --strict, about 1,200 tests on Python 3.11 to 3.14.

The pipeline core has no risk scoring and no opinions. vulnflow.helpers.explore adds optional helpers such as risk_priority(); treat them as a starting point, not a policy.

Installation

vulnflow isn't on PyPI yet. Install it from GitHub:

uv add git+https://github.com/corifeo/vulnflow
pip install git+https://github.com/corifeo/vulnflow

From source, with the example notebooks and Jupyter:

git clone https://github.com/corifeo/vulnflow && cd vulnflow
uv sync --extra notebook

Requires Python 3.11+. HTTP responses are cached under .cache/ in the working directory.

Quick start

from vulnflow.helpers import CacheBuilder

df = CacheBuilder(recent_only=True).build().to_pandas()

That's NVD's Recent feed, the CVEs published in the last eight days, with KEV, EPSS, CWE and CPE already joined and CVSS flattened. The screenshot above is this exact call on 6 November 2026: 3,681 CVEs and 47 columns in a few seconds. The 11% CPE figure is real: NVD's analysis backlog means most new CVEs have no product data yet.

The full library

Once you know what's in the data, build all of it:

from vulnflow.helpers import CacheBuilder

ds = CacheBuilder().ensure("data/nvd.parquet", max_age="1d")

That's every CVE NVD has published, 2002 to today, with the same enrichment. The first call downloads all the yearly feeds and builds the library in about two minutes; after that it reloads from Parquet until the file is a day old.

Feed a warehouse. data/nvd.parquet is the hand-off: one row per CVE, timestamps in UTC, and nested fields such as references and CPE matches as JSON strings. In Snowflake, for example, PUT the file to a stage and COPY INTO a table with MATCH_BY_COLUMN_NAME; PARSE_JSON turns the JSON columns into VARIANT.

Ask it a question. How many CVEs has NVD deferred, meaning it doesn't plan to analyze them?

ds.get_status_distribution()   # {'analyzed': 78669, 'awaiting': 15757, 'deferred': 60950, ...}

deferred = ds.to_pandas(analyzed=False, modified=False, awaiting=False)
deferred["nvd_published_at"].dt.year.value_counts().sort_index()

NVD status counts across all 420,062 CVEs, and Deferred CVEs by year published

On 6 November 2026, 60,950 CVEs (15%) were Deferred, and every one of them was published in 2024 or later: NVD is deferring new CVEs, not old ones. to_pandas() skips rejected CVEs unless you pass rejected=True.

KEV timing in 10 lines

How often does CISA list a CVE as exploited before NVD has even published it? Two years of NVD, joined with the KEV catalog:

import numpy as np
from vulnflow.helpers import CacheBuilder
from vulnflow.transforms.computed import CVDStateTransform
from vulnflow.transforms.inferences import EventInference

ds = CacheBuilder(years=[2024, 2025]).ensure("data/library.parquet", max_age="7d")
kev = ds.filter(np.array([bool(x) for x in ds.get_column("is_kev")]))
kev.transform([EventInference(), CVDStateTransform()])

(kev.get_column("public_to_attack_days") <= 0).mean()   # 0.24

ensure() builds the library once and reloads it from Parquet while it's fresh.

354 KEV CVEs from 2024 and 2025 by when CISA listed them relative to NVD publication; 86 were listed on or before the publication day

Of the 354 CVEs from 2024 and 2025 in KEV, 86 (24%) were listed on or before the day NVD published them. That figure needs only two dates every source agrees on, which brings us to the caveat in What the dates mean. Every number above comes from real data; scripts/make_readme_assets.py regenerates it.

Case study: Log4Shell

from vulnflow import VulnPipeline
from vulnflow.extractors import NVDAPIExtractor
from vulnflow.transforms.computed import CVDStateTransform
from vulnflow.transforms.enrichers import KEVEnricher
from vulnflow.transforms.flatten import KEVFlattenTransform
from vulnflow.transforms.inferences import EventInference

result = VulnPipeline(
    extractors=[NVDAPIExtractor(cve_id="CVE-2021-44228")],
    transformers=[KEVEnricher(), KEVFlattenTransform(), EventInference(), CVDStateTransform()],
).run()
row = result.dataset.to_pandas().iloc[0]

row.cvd_state              # 'VFdPXA': vendor aware, fix ready, public, exploit, attacks
row.is_zero_day_attack     # True: in KEV before the earliest vendor date vulnflow found
row.public_to_attack_days  # -0.43: CISA listed it the day NVD published it

Log4Shell's inferred disclosure timeline and the CVD analytics vulnflow derives from it

tests/integration/test_log4shell.py pins these values against a recorded NVD record, and the screenshot comes from the same record.

Case study: the 2026 surge

Is NVD keeping up? Count the CVEs published by the same date each year, using the full library:

from vulnflow.helpers import CacheBuilder

df = CacheBuilder().ensure("data/nvd.parquet", max_age="1d").to_pandas()
window = df[df["nvd_published_at"].dt.strftime("%m-%d") <= "11-06"]   # 1 Jan to 6 Nov
window.groupby(window["nvd_published_at"].dt.year).agg(
    cves=("cve_id", "size"),
    not_analyzed=("nvd_status", lambda s: s.isin(["awaiting", "deferred"]).mean()),
)
CVEs published by 6 November each year from 2016 to 2026, split into analyzed, awaiting analysis and deferred; 55% of 2026's 92,000 are not analyzed
  • Volume more than doubled. By 6 November, NVD had published 92,067 CVEs in 2026, against 40,343 by the same date in 2025: 2.3 times as many, after growth of under 40% a year since 2018. Monthly volume climbed from 4,302 in January to 17,058 in October.
  • Analysis is falling behind. Until 2023, NVD had analyzed virtually every CVE published by 6 November. The share it hasn't, counting both Awaiting Analysis and Deferred, went from 16% of 2024's CVEs to 35% of 2025's and 55% of 2026's.
  • Most of the gap won't close. Of 2026's 50,936 unanalyzed CVEs, 35,222 are Deferred, meaning NVD doesn't plan to analyze them; 15,714 are still awaiting. Only 45% of 2026's CVEs carry product (CPE) data, against 100% of 2023's.

If you match CVEs to your own products, that last point matters more than the headline count: for most new CVEs, NVD won't tell you what's affected.

What the dates mean

CVD analytics compare six event dates: vendor aware (V), fix ready (F), fix deployed (D), public (P), exploit public (X) and attacks seen (A).

  • Only P and A are source dates. P is NVD's publication date and A is the KEV listing date. Public feeds don't say when a vendor learned about a bug.
  • V, F and X are inferred from NVD reference tags and dates in reference URLs, so they can be wrong. Log4Shell's V comes out as 11 December 2021; Apache actually knew in November.
  • Some events are filled in from others. With no exploit date, X takes the KEV date; with no vendor date but a fix date, V takes the fix date. D (fix deployed) is never inferred, so the state always shows d.
  • No V means unknown, not unaware. It shows as v in the state, and zero-day flags stay False rather than guessing.
  • Date-only sources sort first. KEV and URL dates are midnight UTC, so a same-day KEV listing counts as before a 10:15 publication. A KEV date means attacks were already seen by that day.

public_to_attack_days <= 0 is the most robust signal here, because it uses only P and A.

cvelistV5 instead of NVD

from vulnflow.helpers import CacheBuilder

builder = CacheBuilder(source="cvelist")
ds = builder.ensure("cvelist.parquet", recent=True)  # midnight zip, then today's hourly delta

CVEProject/cvelistV5 is the CVE Program's own record list. It publishes hourly, before NVD, and has the CNA and CISA-ADP scores, CWEs and SSVC directly. NVD keeps the edge on CPE configurations and pre-2024 scoring.

  • Same columns as NVD. nvd_published_at and nvd_modified_at hold datePublished and dateUpdated; nvd_status is published or rejected. metadata has source: "cvelistV5", the assigner, SSVC and the CNA affected list (vendor, product, versions, package URL).
  • CNA scores win. CNA CVSS is authoritative, so the flattened score is the CNA's newest version, then the best ADP score. cvss_source is the CNA or ADP short name.
  • cpes only holds real CPE strings. CNA vendor and product names are not CPE names, so they stay in metadata.
  • Updates. ensure(recent=True) merges today's latest delta (cumulative since midnight UTC). A cache from an earlier day is rebuilt from the newest midnight zip (about 660 MB): a day's last delta stops at 23:00 UTC.
  • Local data. CVEListExtractor(path) reads a release zip or a directory of CVE-*.json.

Validate every record against the CVE JSON schema and business rules with the optional extra:

uv add "vulnflow[cvelist]"
from vulnflow.extractors import CVEListExtractor

ds = CVEListExtractor("2026-11-13_all_CVEs_at_midnight.zip.zip", validate=True).extract()
ds.get_column("cve_valid")             # bool per record
ds.get_column("cve_validation_error")  # e.g. FAILED_JSON_SCHEMA_VALIDATION, or None

Pipelines

from vulnflow import VulnPipeline
from vulnflow.extractors import NVDExtractor
from vulnflow.helpers import CacheBuilder
from vulnflow.loaders import ParquetLoader
from vulnflow.transforms.computed import CVDStateTransform
from vulnflow.transforms.enrichers import KEVEnricher
from vulnflow.transforms.flatten import KEVFlattenTransform
from vulnflow.transforms.inferences import EventInference

pipeline = VulnPipeline(
    extractors=[NVDExtractor(CacheBuilder.feed_url(2025))],
    transformers=[KEVEnricher(), KEVFlattenTransform(), EventInference(), CVDStateTransform()],
    loaders=[ParquetLoader("nvd-2025.parquet")],
)
result = pipeline.run()

result.errors              # failed stages; the run continues past them unless strict=True
result.dataset.to_pandas()

Transforms run in phases computed from their depends_on. Pipelines compose: extract.merge(enrich) combines stages, VulnPipeline.chain([...]) merges a list. VulnDataset.transform() with no arguments runs the registry defaults, for notebooks.

Triage queue export

TriageLoader turns a dataset into a ranked queue for an SSVC triage tool such as VulnTriage: KEV CVEs first, then by EPSS and CVSS, capped at 500 by default.

from vulnflow.helpers import CacheBuilder
from vulnflow.loaders import TriageLoader

ds = CacheBuilder(recent_only=True).build()
TriageLoader("data/triage-queue.json", min_epss=0.01).load(ds)

Each item carries suggested SSVC values with the reason for each. They are heuristics for an analyst to confirm:

  • Exploitation: active if in KEV (or CVSS v4 E:A), poc if NVD tags a reference as Exploit (or E:P), else none.
  • Automatable: yes only with AV:N, AC:L, PR:N and UI:N (CVSS v4's AU metric wins when present).
  • Technical impact: total with high confidentiality and integrity impact, else partial.

Mission and well-being are yours to set: no public feed knows your assets.

Sources and outputs

Stage Available
Extract NVD JSON feeds (files or URLs), NVD REST API 2.0, cvelistV5 (release zips or a directory), MSRC CVRF (Patch Tuesday), Parquet, JSON, pandas, polars
Enrich CISA KEV, EPSS, CWE, CAPEC, NVD change history
Flatten CVSS (v2, v3.x, v4), CPE, CWE, CAPEC, KEV, EPSS, STRIDE, MSRC
Infer CVD event timestamps, tags, STRIDE categories
Compute CVD state, zero-day and exposure flags, desiderata, time deltas
Load Parquet, JSON, pandas, polars, SSVC triage queue (JSON)

Extractors live in vulnflow.extractors, loaders in vulnflow.loaders, and transforms in vulnflow.transforms.enrichers, .flatten, .inferences and .computed.

Examples & notebooks

01-first-look.ipynb the first five minutes: Recent feed to pandas
02-build-library.ipynb a persistent Parquet library with incremental updates
03-weekly-triage.ipynb a Monday triage of what changed this week
04-patch-tuesday.ipynb NVD API queries and Microsoft Patch Tuesday
05-threat-landscape.ipynb trends and charts across 2024 and 2025
06-custom-pipeline.ipynb building and composing VulnPipelines
examples/scripts/ cron refresh, a KEV alert, a CI pipeline

Setup and troubleshooting: examples/README.md.

Related

VulnState models a single CVE's disclosure lifecycle as a state machine. vulnflow is the data pipeline: it computes the same CVD state at scale from public feeds.

Development

uv sync
uv run pytest
uv run ruff check src/ tests/
uv run mypy src/ --strict
uv run black --check src/ tests/
uv run --with rich --extra notebook python scripts/make_readme_assets.py   # regenerate README images

Issues and PRs are welcome, especially new extractors and enrichers. Each one is a small class; see the existing ones in src/vulnflow/ and CONTRIBUTING.md. To report a security issue, see SECURITY.md.

Citation

To cite vulnflow, use Cite this repository on GitHub (CITATION.cff).

Data sources and notices

vulnflow downloads data at run time from its publishers; it ships none of it. Each source's terms apply to what you download.

  • This product uses data from the NVD API but is not endorsed or certified by the NVD.
  • CVE records from the CVE Program's cvelistV5 repository (CVE is a registered trademark of The MITRE Corporation).
  • CISA Known Exploited Vulnerabilities catalog, from the Cybersecurity and Infrastructure Security Agency.
  • EPSS scores from FIRST and Empirical Security.
  • CWE and CAPEC are trademarks of The MITRE Corporation.
  • Microsoft Security Response Center CVRF API.

License

MIT

About

Fast, boring ETL for public vulnerability data: NVD, CISA KEV, EPSS, CWE, CAPEC and Microsoft Patch Tuesday in one columnar dataset for pandas, polars, Parquet or your warehouse.

Topics

Resources

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Contributors

Languages