cmem-plugin-kafka

This eccenca Corporate Memory plugin allows for sending and receiving messages from Apache Kafka.

Installation

cmemc -c my-cmem admin workspace python install cmem-plugin-kafka

Development

• Run task to see all major development tasks.
• Use pre-commit to avoid errors before commit.
• Agent instructions and skills for this project are in .claude/ - your own instructions belong in CLAUDE.md, which is never overwritten.
• This repository was created with this copier template.

Running Test

This plugin needs a reachable Corporate Memory instance. Kafka is provided automatically by the test suite itself (via testcontainers, which needs a running local Docker daemon) - no manual Kafka setup or .env entries are needed for it.

In order to setup access to your Corporate Memory uses cmemc's config eval command to fill environment variables:

$ eval $(cmemc -c my-cmem config eval)

Having Corporate Memory in place and Docker running, run the test suite with task check.

confluent-python installation

ARM based Macs

• make sure brew doctor has no issues
• install with brew install librdkafka
• Provide this in your environment (based on this answer)

export CPATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib

• test build from source in separate environment with:
  • pip install https://files.pythonhosted.org/packages/fb/16/d04dded73439266a3dbcd585f1128483dcf509e039bacd93642ac5de97d4/confluent-kafka-1.8.2.tar.gz
• then try poetry install

Kafka CLI Utility

To manually poke at a broker outside of a test run (e.g. with the CLI tool below), you can start one via task kafka:start (and stop it again with task kafka:stop) - it brings up docker/docker-compose.yml on localhost:9093, independent of the test suite's own testcontainers-managed broker.

kcat (formerly kafkacat) is a command-line utility that you can use to test and debug Apache Kafka® deployments. kcat is an open-source utility, available at https://github.com/edenhill/kcat.

To send messages

docker run -it --rm \
        edenhill/kcat \
                -b kafka-broker:9092 \
                -t test \
                -K: \
                -P 

To consume messages

ocker run -it --rm \
        edenhill/kcat \
           -b kafka-broker:9092 \
           -C \
           -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\tOffset: %o\n--\n' \
           -t test