ColonyOS is an open-source framework for seamless execution of computational workloads across heterogeneous platforms - cloud, edge, HPC, IoT devices, and beyond. It creates Compute Continuums by providing a unified orchestration layer that operates as a meta-orchestrator on top of existing infrastructure.
Traditional orchestration systems are tied to specific platforms (Kubernetes for cloud, Slurm for HPC, etc.). ColonyOS breaks these silos through meta-process management - a broker-based architecture that separates computational intent from execution.
Example of use cases:
- Scientific Computing: Process satellite imagery, analyze sensor data, run simulations across HPC clusters
- AI/ML Pipelines: Distribute training jobs, run inference on edge devices, orchestrate multi-agent LLM systems
- Serverless at Scale: Build FaaS platforms that span cloud, edge, and on-premise infrastructure
- Data Processing: ETL pipelines, batch processing, real-time stream processing with ColonyFS integration
- Industrial IoT: Coordinate computations across factory floor devices, edge gateways, and cloud
- Earth Observation: Automated satellite image processing and analysis workflows
- Infrastructure as Code: Declaratively manage infrastructure across computing continuums - define services spanning cloud, edge, HPC, and IoT with GitOps workflows, automatic drift detection, and self-healing reconciliation
Declarative Intent + Broker + Distributed Execution = Computing Continuums
Instead of writing platform-specific code, you declare WHAT you want to compute using Function Specifications. The Colonies Server acts as a broker that matches your intent with available Executors (distributed workers) that know HOW to execute on their specific platforms. This separation creates seamless Computing Continuums across heterogeneous infrastructure.
- Platform Agnostic: Same function specification runs on Kubernetes, HPC, edge devices, IoT - executors translate to platform-specific execution
- Decoupled Architecture: Submit work anytime, execute asynchronously - temporal and spatial decoupling via broker
- Zero-Trust by Design: No session tokens, no passwords - every request cryptographically signed with ECDSA (secp256k1)
- Protocol Flexibility: HTTP/REST API with WebSocket support for real-time communication
- Pull-Based Execution: Executors connect from anywhere (even behind NAT/firewalls) and pull work - no need for inbound access
- Built-in Audit Trail: Every execution recorded as an immutable ledger for compliance and debugging
- Real-Time Reactive: WebSocket subscriptions for instant notifications on workflow state changes
- HTTP/REST Architecture: Clean REST API with WebSocket support for real-time subscriptions
- Distributed Execution: Executors run anywhere on the Internet - supercomputers, edge devices, browsers, embedded systems
- Zero-Trust Security: All communication cryptographically signed with ECDSA (secp256k1)
- Workflow DAGs: Complex computational pipelines with parent-child dependencies
- Event-Driven: Real-time WebSocket subscriptions for process state changes
- Scheduled Execution: Cron-based and interval-based job scheduling
- Dynamic Batching: Generators that pack arguments and trigger workflows based on counter or timeout conditions
- Blueprint Reconciliation: Kubernetes-style declarative deployment management with automatic drift detection and correction
- Modular Server: A small orchestration kernel plus config-enabled modules - disable what you do not use
- Full Audit Trail: Complete execution history stored as an immutable ledger
- High Availability: Etcd-based clustering with automatic failover
- Multi-Language SDKs: Go, Rust, Python, Julia, JavaScript, Haskell
- Colony: A distributed runtime environment - a network of loosely connected Executors
- Executor: Distributed worker that pulls and executes workloads (can be implemented in any language, runs anywhere)
- Process: Computational workload with states: WAITING → RUNNING → SUCCESS/FAILED
- FunctionSpec: Specification defining what computation to run and execution conditions
- ProcessGraph: Workflow represented as a Directed Acyclic Graph (DAG)
- Blueprint: Declarative infrastructure specification with desired state management
- Reconciliation: Automatic drift detection and correction that maintains blueprints in their desired state
- Submit: Users submit function specifications to the Colonies server
- Schedule: The scheduler assigns processes to available Executors based on conditions
- Execute: Executors pull assigned processes, execute them, and report results
- Chain: Complex workflows span multiple platforms by chaining processes together
- Monitor: Real-time subscriptions and full execution history enable observability
Colonies implements a zero-trust architecture where all communication is cryptographically signed:
- No traditional authentication tokens or session management
- Each request signed with an ECDSA (secp256k1) private key; the server recovers the caller identity directly from the signature
- Server validates signatures and enforces role-based access control
- Executors can operate on untrusted infrastructure while maintaining security
Run a complete ColonyOS server, with an embedded PostgreSQL, in a single container:
docker run -d --name colonies -p 50080:50080 \
-v colonies-data:/var/lib/postgresql/data \
colonyos/colonies-allinone:latest
docker logs colonies # prints the generated server, colony, and user keysThe container generates its identity keys on first start and persists them, together with the database, in the colonies-data volume. The same setup is available as a Compose profile (docker compose --profile allinone up), and the repository's docker-compose.yml also provides scale-up profiles: postgres (separate PostgreSQL), fs (adds SeaweedFS-backed file storage), and ha (three-server cluster). See the Deployment Guide.
Configuration is environment-driven with an optional config file; server features are grouped into config-enabled modules. See Configuration and Modules, and use colonies config check to validate a setup.
Comprehensive step-by-step tutorials are available in the tutorials repository:
The Colonies Dashboard provides a web UI for monitoring and managing your compute continuum:
- Installation Guide - Install and configure Colonies
- Getting Started - Your first Colonies application
- Configuration - Environment variables and settings
- Introduction - Core concepts and architecture
- Implementing Executors - Create executors in Python, Go, Julia, JavaScript
- Fibonacci Tutorial (Go) - Complete example application
- Workflow DAGs - Create complex computational pipelines
- Generators - Batch processing and dynamic workflows
- Cron Jobs - Schedule recurring tasks
- Blueprints - Declarative, reconciled deployments
- Channels Tutorial - Bidirectional communication with running processes
- CLI Usage - Command-line interface reference
- Logging - Process logging and monitoring
- Modules - The orchestration kernel and config-enabled modules
- RPC Protocol - HTTP RPC protocol specification
- Security Design - Zero-trust security model
- Channels Design - Channel internals and limits
- Schema Validation - Blueprint schema validation
- Reconciliation - Blueprint reconciliation internals
- Deployment Guide - Single-container quickstart and scale-up profiles
- Container Building - Build Docker containers for single and multi-platform
- High-Availability Deployment - Production cluster setup
- Monitoring - Grafana and Prometheus integration
- Migrating from v1.x - v2.0 migration guide and deprecation policy
- Kubernetes Helm Charts - Deploy on Kubernetes
- Go SDK - Official Go client library
- Python SDK - Python client library
- Rust SDK - Rust client library
- Julia SDK - Julia client library
- JavaScript SDK - JavaScript/Node.js library
- Haskell SDK - Haskell client library
- Executors - Pre-built executor implementations
The repository contains a development container configuration to simplify development environment setup. You can use it locally or in a GitHub Codespace. It is a single Go container with Docker-in-Docker; the post-create step builds the binary, starts throwaway PostgreSQL and SeaweedFS (S3) backends for integration tests via make dev-up, and generates a local .env with fresh keys. Unit tests need no services; no further configuration is required.
Local Development (VS Code):
- Install Docker on your machine.
- Install the Dev Containers extension in Visual Studio Code.
- Clone this repository.
- Open the folder in Visual Studio Code.
- When prompted, select "Reopen in Container" or use the command "Dev Containers: Open Folder In Container..." from the command palette.
GitHub Codespaces:
- Simply create a Codespace from the repository page on GitHub. The development container will be set up automatically.
make build # Build the main colonies binary
make container # Build Docker container for local architecture
make container-multiplatform # Build for amd64 and arm64
make install # Install to /usr/local/binFor detailed instructions on building containers including multi-platform builds, see the Container Building Guide.
make test # Unit tests only, no services required (seconds)
make dev-up # Start Postgres + SeaweedFS backends for integration tests
make test-integration # Unit + integration tests (needs the dev-up backends)
make test-compat # Frozen wire-API compatibility suitemake coverage # Generate coverage reportsColonyOS is currently used in production by:
- RockSigma AB - Automatic seismic processing engine for underground mines, orchestrating workloads across cloud and edge infrastructure
Contributions are welcome! Please see our contributing guidelines and code of conduct.
- Website: colonyos.io
- GitHub: github.com/colonyos
- Tutorials: github.com/colonyos/tutorials
See LICENSE file for details.