OSSIP

A hub for Open Source Software Improvement Proposals

ossip.dev

This repo holds scripts for generating a website which collects and enriches the Improvement Proposals (IPs) from various open source software (OSS) projects (hence the name).

Currently supported projects are:

• Apache Kafka project's Kafka Improvement Proposals (KIPs)
• Apache Flink project's Flink Improvement Proposals (FLIPs)
• Strimzi project's Strimzi Improvement Proposals (SIPs)
• StreamsHub project's StreamsHub Improvement Proposals (SHIPs)
• Kroxylicious project's Kroxylicious Design Proposals (KDPs)

The Kafka and Flink pipelines read from Apache Confluence wikis and mailing list archives; the Strimzi, StreamsHub and Kroxylicious pipelines read proposal PRs from GitHub repositories.

JSON API and OSSIP Skill

The OSSIP project provides a JSON API for programmatic access to KIP and FLIP data. The API is:

• Refreshed daily from wiki and mailing list archives (approximately 09:30 UTC)
• CORS-enabled for direct browser access
• Versioned under /api/v1/ for stable integration

API Endpoints

• https://ossip.dev/api/v1/index.json — API entry point
• https://ossip.dev/api/v1/kafka/kips.json — All Kafka KIPs (summary)
• https://ossip.dev/api/v1/kafka/kips/{id}.json — Individual KIP detail
• https://ossip.dev/api/v1/flink/flips.json — All Flink FLIPs (summary)
• https://ossip.dev/api/v1/flink/flips/{id}.json — Individual FLIP detail
• https://ossip.dev/api/v1/strimzi/sips.json — All Strimzi SIPs (summary)
• https://ossip.dev/api/v1/strimzi/sips/{id}.json — Individual SIP detail
• https://ossip.dev/api/v1/streamshub/ships.json — All StreamsHub SHIPs (summary)
• https://ossip.dev/api/v1/streamshub/ships/{id}.json — Individual SHIP detail
• https://ossip.dev/api/v1/kroxylicious/kdps.json — All Kroxylicious KDPs (summary)
• https://ossip.dev/api/v1/kroxylicious/kdps/{id}.json — Individual KDP detail
• https://ossip.dev/api/v1/schemas/ — JSON Schema definitions

OSSIP Skill for AI Agents

The OSSIP skill enables AI agents (Claude Code, Claude Desktop, etc.) to query the JSON API directly. Install it by downloading https://ossip.dev/skill/ossip/SKILL.md to your agent's skills directory:

Claude Code (project-scoped):

mkdir -p .claude/skills/ossip
curl -o .claude/skills/ossip/SKILL.md https://ossip.dev/skill/ossip/SKILL.md

Claude Code (personal/global):

mkdir -p ~/.claude/skills/ossip
curl -o ~/.claude/skills/ossip/SKILL.md https://ossip.dev/skill/ossip/SKILL.md

For full documentation, see ossip.dev/api.html.

Development

Installation

This project uses uv to manage dependencies. To install the necessary libraries run:

$ uv sync

Running Tests

This project includes unit tests for critical functions. To run the test suite:

# Run all tests
$ uv run pytest

# Run with verbose output
$ uv run pytest -v

# Run specific test file
$ uv run pytest tests/common/test_utils.py

# Run tests without coverage report
$ uv run pytest --no-cov

Current test coverage focuses on critical functions in the ipper/common module:

• utils.py: 100% coverage (date/time utilities)
• mailing_list.py: 40% coverage (parsing functions)
• Overall ipper/common: 35% coverage

The test suite includes 38 tests covering:

• Month list generation (prevents future month bugs)
• Email timestamp parsing (multiple formats)
• Vote parsing (binding votes only)
• Metadata handling
• Age calculation

Coverage reports are generated in HTML format (see htmlcov/index.html) and in the terminal output.

Code Quality Checks

This project uses Ruff for fast linting and formatting:

# Check code for issues
$ uv run ruff check ipper/

# Auto-fix issues (removes unused imports, fixes formatting, etc.)
$ uv run ruff check ipper/ --fix

# Format code
$ uv run ruff format ipper/

# Check formatting without modifying files
$ uv run ruff format --check ipper/

Ruff checks for:

• Unused imports (auto-removed)
• Import sorting (auto-organized)
• Modern Python idioms (e.g., list[str] instead of List[str])
• Common bugs (mutable defaults, etc.)
• Code simplifications

Configuration is in pyproject.toml under [tool.ruff].

Type checking with MyPy:

# Run type checker
$ uv run mypy ipper/

Downloading and processing KIP data

To download the Apache Kafka dev mailing list for the last year (longer periods can be set via the --days option), process the archives and download the KIP Wiki information from the confluence site; run the init command:

$ uv run python ipper/main.py kafka init --days 365

To update only the most recent month and add any new KIPs which have been posted since the last update run:

$ uv run python ipper/main.py kafka update

To reprocess ALL Kafka mbox files from scratch (useful when processing logic changes or cache is corrupted):

$ uv run python ipper/main.py kafka refresh

Downloading and processing FLIP data

To download the Flink dev mailing list for the last year, process the archives and download the FLIP Wiki information from the confluence site; run the init command:

$ uv run python ipper/main.py flink init --days 365

To update only the most recent month and add any new FLIPs which have been posted since the last update run:

$ uv run python ipper/main.py flink update

To reprocess ALL Flink mbox files from scratch (useful when processing logic changes or cache is corrupted):

$ uv run python ipper/main.py flink refresh

Downloading and processing SIP/SHIP/KDP data (GitHub projects)

Strimzi, StreamsHub and Kroxylicious track their proposals with pull requests on GitHub, so there is no wiki/mail pipeline - just proposal data and output.

init and refresh do a full fetch from GitHub and require a GITHUB_TOKEN environment variable (a full Strimzi fetch costs ~500 requests against the 60/hour unauthenticated limit):

$ export GITHUB_TOKEN=ghp_...   # https://github.com/settings/tokens
$ uv run python ipper/main.py strimzi init
$ uv run python ipper/main.py streamshub init
$ uv run python ipper/main.py kroxylicious init

update is incremental (typically ~5-15 requests per project per run) and works unauthenticated:

$ uv run python ipper/main.py strimzi update
$ uv run python ipper/main.py streamshub update
$ uv run python ipper/main.py kroxylicious update

To reprocess from scratch (same as init):

$ uv run python ipper/main.py strimzi refresh

Building the site

Quick Local Build

For convenience, a local_build.sh script is provided that automates the entire build process:

# Full build (updates data + generates HTML)
$ ./local_build.sh

# Render-only build (skips data updates, only regenerates HTML from cached data)
$ ./local_build.sh --render-only

Manual Build Steps

To create the standalone Kafka site html run the command below where the first argument is the kip mentions cache file produced by the step above and the second is the html output filepath:

$ uv run python ipper/main.py kafka output standalone cache/mailbox_files/kip_mentions.csv site_files/kafka.html

To create the Flink site html with individual FLIP pages:

$ uv run python ipper/main.py flink output cache/flip_wiki_cache.json site_files/flink.html site_files/flips

To create the GitHub-project site html with individual detail pages (example for Strimzi; use streamshub/ships and kroxylicious/kdps for the others):

$ uv run python ipper/main.py strimzi output cache/sip_proposals_cache.json site_files/strimzi.html site_files/sips

This generates a main index page and individual detail pages in the specified output directory. Pass --api-dir site_files/api/v1/strimzi to also emit the JSON API files.

You will also need to copy over the static files from the templates directory to the site_files directory:

$ mkdir -p site_files/assets
$ cp templates/index.html site_files
$ cp templates/style.css site_files
$ cp -r templates/assets site_files/assets

Deployment

A Github action (see .github/publish.yaml) will build and publish the site on every push to main. The site is automatically built and deployed every day at approximately 09:30 UTC.