From c9bae9987f10acc926b2767983044a301785f7a4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Wed, 4 Feb 2026 11:39:07 +0100 Subject: [PATCH 1/5] feat: add Docker branch isolation for parallel development Implement automatic environment isolation based on git branch names, allowing developers to work on multiple branches simultaneously without conflicts between databases, containers, or Docker volumes. Key changes: - Auto-detect git branch and normalize name (replace special chars) - Tag Docker images by branch (python.ie/website:{branch}) - Isolate PostgreSQL databases per branch (pythonie_{branch}) - Create separate Docker volumes per branch - Add configurable ports for parallel development (WEB_PORT, PG_PORT) - Upgrade Redis from 6.2 to 7-alpine with persistence - Add healthchecks for PostgreSQL and Redis - Remove unused minio and mc services New Taskfile commands: - task branch:info - Show current branch environment - task branch:volumes - List all Docker volumes by branch - task branch:clean - Remove volumes for current branch - task env:write - Generate .env file for docker-compose Documentation: - Add DOCKER-BRANCHES.md with usage examples and troubleshooting - Add .env.example template for custom port overrides - Update .gitignore to exclude generated .env file This enables developers to switch branches or run multiple instances in parallel without losing data or experiencing port conflicts. Co-Authored-By: Claude Sonnet 4.5 --- .env.example | 16 ++++ .gitignore | 1 + DOCKER-BRANCHES.md | 223 +++++++++++++++++++++++++++++++++++++++++++++ Taskfile.yaml | 69 +++++++++++++- docker-compose.yml | 74 ++++++++------- 5 files changed, 344 insertions(+), 39 deletions(-) create mode 100644 .env.example create mode 100644 DOCKER-BRANCHES.md diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..a0635f1 --- /dev/null +++ b/.env.example @@ -0,0 +1,16 @@ +# Docker Compose Environment Variables +# Copy this file to .env and customize as needed + +# Override ports for parallel development (optional) +# Useful when running multiple branches simultaneously +# WEB_PORT=8001 +# PG_PORT=5433 + +# Override git branch detection (optional - normally auto-detected) +# GIT_BRANCH=my-custom-branch + +# Override Docker image name (optional) +# DOCKER_IMAGE=python.ie/website:custom-tag + +# Override database name (optional) +# PGDATABASE=pythonie_custom diff --git a/.gitignore b/.gitignore index d96d0d2..56ecc1b 100644 --- a/.gitignore +++ b/.gitignore @@ -66,6 +66,7 @@ project.db # Environment files with sensitive credentials .envrc +.env development.env production.env diff --git a/DOCKER-BRANCHES.md b/DOCKER-BRANCHES.md new file mode 100644 index 0000000..4ff58b0 --- /dev/null +++ b/DOCKER-BRANCHES.md @@ -0,0 +1,223 @@ +# Docker Multi-Branch Development + +This setup allows you to run isolated Docker environments for each git branch, preventing conflicts when switching branches or working on multiple features simultaneously. + +## How It Works + +### Automatic Branch Detection + +The system automatically: +- Detects your current git branch (`git rev-parse --abbrev-ref HEAD`) +- Normalizes the branch name (e.g., `feature/add-auth` → `feature-add-auth`) +- Uses this normalized name for: + - Docker image tags: `python.ie/website:feature-add-auth` + - Database names: `pythonie_feature-add-auth` + - Container names: `pythonie-web-feature-add-auth` + - Docker volumes: `postgres-data-feature-add-auth` + +### Branch Isolation + +Each branch gets: +- ✅ Its own Docker image +- ✅ Its own PostgreSQL database (isolated data) +- ✅ Its own Redis instance (isolated cache) +- ✅ Its own Docker volumes (persistent storage) +- ✅ Unique container names (no conflicts) + +## Usage + +### Basic Commands + +```bash +# Check your current branch environment +task branch:info + +# Build image for current branch +task docker:build + +# Run development server (uses current branch) +task run + +# Run tests (uses current branch database) +task tests + +# Django shell for current branch +task django:shell-plus +``` + +### Working on Multiple Branches + +**Scenario**: You want to work on `feature/auth` while keeping `main` running. + +```bash +# Terminal 1: Main branch +git checkout main +task run +# → Uses: python.ie/website:main, pythonie_main database, port 8000 + +# Terminal 2: Feature branch (in another terminal) +git checkout feature/auth +task run +# → ERROR: Port 8000 already in use! +``` + +**Solution**: Use custom ports for parallel instances: + +```bash +# Terminal 1: Main branch +git checkout main +task run + +# Terminal 2: Feature branch with custom port +git checkout feature/auth +WEB_PORT=8001 PG_PORT=5433 task run +# → Uses: port 8001, separate database +``` + +### Managing Volumes + +```bash +# List all volumes (all branches) +task branch:volumes + +# Example output: +# pythonie-postgres-data-master +# pythonie-postgres-data-feature-add-auth +# pythonie-redis-data-master +# pythonie-redis-data-feature-add-auth + +# Clean volumes for current branch (DESTRUCTIVE!) +task branch:clean +# → Deletes all data for current branch +``` + +### Switching Branches + +When you switch branches, the system automatically uses the correct environment: + +```bash +# On main branch +git checkout main +task run # Uses main database + +# Switch to feature branch +git checkout feature/add-auth +task run # Automatically uses feature-add-auth database +``` + +**Important**: You must rebuild the image after switching branches if dependencies changed: + +```bash +git checkout feature/new-dependencies +task docker:build # Rebuild image with new dependencies +task run +``` + +## Advanced: Custom Ports for Parallel Development + +Create a `.env` file (git-ignored) to override default ports: + +```bash +# .env +WEB_PORT=8001 +PG_PORT=5433 +``` + +Or pass them inline: + +```bash +WEB_PORT=8001 PG_PORT=5433 task run +``` + +## Troubleshooting + +### Port Already in Use + +``` +Error: port 8000 already in use +``` + +**Solution**: Stop other instances or use custom ports: +```bash +task down # Stop all containers for current branch +# OR +WEB_PORT=8001 PG_PORT=5433 task run +``` + +### Database Not Found + +``` +FATAL: database "pythonie_feature-xyz" does not exist +``` + +**Solution**: Run migrations to create the database: +```bash +task django:migrate +``` + +### Wrong Database Being Used + +```bash +# Check which environment is active +task branch:info + +# Ensure you're on the correct git branch +git branch + +# Rebuild if needed +task docker:build +``` + +### Cleaning Up Old Branches + +After deleting a git branch, clean up its Docker resources: + +```bash +# Remove unused images +docker image prune -a + +# Remove volumes for deleted branches +# (replace 'old-branch-name' with actual normalized branch name) +docker volume rm pythonie-postgres-data-old-branch-name +docker volume rm pythonie-redis-data-old-branch-name + +# Or use the helper (must be on that branch) +git checkout old-branch-name +task branch:clean +``` + +## Environment Variables Reference + +| Variable | Default | Description | +|----------|---------|-------------| +| `GIT_BRANCH` | Auto-detected | Current git branch (normalized) | +| `DOCKER_IMAGE` | `python.ie/website:{branch}` | Docker image name | +| `PGDATABASE` | `pythonie_{branch}` | PostgreSQL database name | +| `WEB_PORT` | `8000` | Web server port | +| `PG_PORT` | `5432` | PostgreSQL port | + +## Tips + +1. **Branch Naming**: Use descriptive branch names. They'll appear in container names and logs. + +2. **Disk Space**: Each branch creates separate volumes. Monitor disk usage: + ```bash + docker system df + ``` + +3. **Production**: This system is for **development only**. Production uses fixed names and ports. + +4. **Database Migrations**: Each branch has independent migrations. Remember to run `task django:migrate` after switching branches with schema changes. + +5. **Shared Data**: If you need to copy data between branches: + ```bash + # Export from main + git checkout main + task run:postgres # Start just postgres + pg_dump -U postgres pythonie_main > /tmp/main.sql + + # Import to feature branch + git checkout feature/xyz + task run:postgres + psql -U postgres pythonie_feature-xyz < /tmp/main.sql + ``` diff --git a/Taskfile.yaml b/Taskfile.yaml index 9781d74..e4e72ad 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -3,13 +3,26 @@ version: '3' +vars: + # Detect current git branch and normalize the name + # (replace / with - and remove special characters) + GIT_BRANCH: + sh: git rev-parse --abbrev-ref HEAD 2>/dev/null | sed 's/[^a-zA-Z0-9._-]/-/g' | tr '[:upper:]' '[:lower:]' || echo "dev" + env: - DOCKER_IMAGE: python.ie/website-dev - PGDATABASE: pythonie + # Pass branch name to docker-compose + GIT_BRANCH: "{{.GIT_BRANCH}}" + # Docker image tagged by branch + DOCKER_IMAGE: "python.ie/website:{{.GIT_BRANCH}}" + # Database isolated by branch + PGDATABASE: "pythonie_{{.GIT_BRANCH}}" PGPASSWORD: pythonie PGUSER: postgres PGHOST: 127.0.0.1 HEROKU_APP: pythonie + # Dynamic ports (optional - can stay fixed if running only one instance at a time) + WEB_PORT: 8000 + PG_PORT: 5432 dotenv: ['production.env'] @@ -100,48 +113,68 @@ tasks: cmds: - heroku maintenance:off + env:write: + desc: Write environment variables to .env file for docker-compose + cmds: + - | + cat > .env </dev/null || echo "Postgres volume not found" + - docker volume rm pythonie-redis-data-{{.GIT_BRANCH}} 2>/dev/null || echo "Redis volume not found" + - echo "Volumes cleaned for branch {{.GIT_BRANCH}}" diff --git a/docker-compose.yml b/docker-compose.yml index 702d522..d3c282d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,68 +1,72 @@ services: web: - image: python.ie/website-dev + build: . + image: "python.ie/website:${GIT_BRANCH:-dev}" + container_name: "pythonie-web-${GIT_BRANCH:-dev}" ports: - - "8000:8000" + - "${WEB_PORT:-8000}:8000" command: /usr/bin/fish depends_on: - - postgres - - minio - - redis + postgres: + condition: service_healthy + redis: + condition: service_healthy env_file: - development.env environment: -# DJANGO_SETTINGS_MODULE: pythonie.settings.dev - PGDATABASE: pythonie + PGDATABASE: "pythonie_${GIT_BRANCH:-dev}" PGUSER: postgres PGPASSWORD: pythonie + GIT_BRANCH: "${GIT_BRANCH:-dev}" volumes: - .:/app working_dir: /app - postgres: image: postgres:17 - ports: - - "5432:5432" + container_name: "pythonie-postgres-${GIT_BRANCH:-dev}" + # ports: + # - "${PG_PORT:-5432}:5432" environment: POSTGRES_PASSWORD: pythonie + POSTGRES_DB: "pythonie_${GIT_BRANCH:-dev}" + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres"] + interval: 10s + timeout: 5s + retries: 5 + restart: unless-stopped + volumes: + - postgres-data:/var/lib/postgresql/data redis: - image: redis:6.2 - - minio: - image: quay.io/minio/minio - ports: - - "9000:9000" - - "9001:9001" - command: server --console-address ":9001" /data - environment: - MINIO_ROOT_USER: pythonie - MINIO_ROOT_PASSWORD: pythonie + restart: unless-stopped + image: redis:7-alpine + container_name: "pythonie-redis-${GIT_BRANCH:-dev}" volumes: - - minio-data:/data + - redis-data:/data + command: redis-server --appendonly yes healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] - interval: 30s - timeout: 20s + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 3s retries: 3 - mc: - image: quay.io/minio/mc - links: - - minio - volumes: - - ./mc:/root/.mc - - ../s3:/s3 - test: - image: python.ie/website-dev + image: "python.ie/website:${GIT_BRANCH:-dev}" environment: DJANGO_SETTINGS_MODULE: pythonie.settings.tests + PGDATABASE: "pythonie_${GIT_BRANCH:-dev}" + depends_on: + postgres: + condition: service_healthy volumes: - .:/app working_dir: /app command: python pythonie/manage.py test pythonie --verbosity=3 volumes: - minio-data: + postgres-data: + name: "pythonie-postgres-data-${GIT_BRANCH:-dev}" + redis-data: + name: "pythonie-redis-data-${GIT_BRANCH:-dev}" From cc17f4773ebe531fa015d26681f01c4973a3329d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Wed, 4 Feb 2026 12:09:23 +0100 Subject: [PATCH 2/5] feat: add SQLite database isolation by branch and documentation Extend branch isolation to SQLite databases for local development and add comprehensive documentation for the branch isolation feature. Changes to SQLite isolation: - Modify dev.py to use branch-specific SQLite files (db-{branch}.sqlite3) - Add *.sqlite3 to .gitignore to exclude database files - Remove debug print statement from dev.py - Each branch now has isolated SQLite data without requiring worktrees Documentation improvements: - Add complete "Docker Branch Isolation" section to README.md - Document all branch:* commands (info, verify, volumes, clean) - Add examples for multi-branch development and branch switching - Add troubleshooting section for database configuration issues - Update DOCKER-BRANCHES.md to mention SQLite isolation - Add note in local setup about GIT_BRANCH export New Taskfile command: - task branch:verify - Comprehensive verification of branch configuration Shows git branch, expected database, .env file, and PostgreSQL databases This ensures complete data isolation between branches whether using Docker (PostgreSQL) or local development (SQLite), preventing data loss when switching branches. Co-Authored-By: Claude Sonnet 4.5 --- .gitignore | 1 + DOCKER-BRANCHES.md | 1 + README.md | 85 ++++++++++++++++++++++++++++++- Taskfile.yaml | 13 +++++ pythonie/pythonie/settings/dev.py | 4 +- 5 files changed, 102 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 56ecc1b..a86116c 100644 --- a/.gitignore +++ b/.gitignore @@ -73,6 +73,7 @@ production.env # Database dumps and local databases *.dump *.duckdb +*.sqlite3 # Local settings pythonie/pythonie/settings/pgdev.py diff --git a/DOCKER-BRANCHES.md b/DOCKER-BRANCHES.md index 4ff58b0..cef9158 100644 --- a/DOCKER-BRANCHES.md +++ b/DOCKER-BRANCHES.md @@ -20,6 +20,7 @@ The system automatically: Each branch gets: - ✅ Its own Docker image - ✅ Its own PostgreSQL database (isolated data) +- ✅ Its own SQLite database for local dev: `pythonie/db-{branch}.sqlite3` - ✅ Its own Redis instance (isolated cache) - ✅ Its own Docker volumes (persistent storage) - ✅ Unique container names (no conflicts) diff --git a/README.md b/README.md index ba51e72..a3f2d45 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 ## Quick Start (Docker - Recommended) +**Note**: Each git branch automatically gets its own isolated database and Docker environment. See [Docker Branch Isolation](#docker-branch-isolation) below. + 1. Build the Docker image: ```bash task docker:build @@ -20,7 +22,7 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 2. Start supporting services: ```bash - docker compose up -d postgres redis minio + docker compose up -d postgres redis ``` 3. Run database migrations: @@ -48,10 +50,79 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 7. Visit http://127.0.0.1:8000/ to see the site with sample content 8. Access Wagtail admin at http://127.0.0.1:8000/admin/ +## Docker Branch Isolation + +The project automatically isolates Docker environments by git branch, allowing you to work on multiple branches simultaneously without conflicts. + +### How It Works + +Each git branch gets: +- **Separate Docker image**: `python.ie/website:{branch-name}` +- **Isolated PostgreSQL database**: `pythonie_{branch-name}` (for docker-compose with pgdev settings) +- **Isolated SQLite database**: `pythonie/db-{branch-name}.sqlite3` (for local dev) +- **Dedicated volumes**: `pythonie-postgres-data-{branch}`, `pythonie-redis-data-{branch}` +- **Unique containers**: `pythonie-web-{branch}`, `pythonie-postgres-{branch}`, `pythonie-redis-{branch}` + +### Branch Commands + +```bash +# Check your current branch environment +task branch:info + +# Verify database configuration +task branch:verify + +# List all Docker volumes (all branches) +task branch:volumes + +# Clean up volumes for current branch (DESTRUCTIVE!) +task branch:clean +``` + +### Working on Multiple Branches + +**Scenario**: Work on `main` and `feature/new-auth` simultaneously. + +```bash +# Terminal 1: Main branch on default ports +git checkout main +task run # Uses pythonie_main database on port 8000 + +# Terminal 2: Feature branch on custom ports +git checkout feature/new-auth +WEB_PORT=8001 PG_PORT=5433 task run # Uses pythonie_feature-new-auth on port 8001 +``` + +### Switching Branches + +When you switch branches, the system automatically uses the correct environment: + +```bash +git checkout main +task run # Automatically uses main database + +git checkout feature/xyz +task run # Automatically uses feature-xyz database (isolated from main) +``` + +**Note**: Rebuild the Docker image after switching branches if dependencies changed: +```bash +git checkout feature/new-dependencies +task docker:build +task run +``` + +For detailed documentation, see [DOCKER-BRANCHES.md](DOCKER-BRANCHES.md). + ## Local Setup (Without Docker) If you prefer to develop without Docker: +**Note**: When developing locally, each git branch automatically uses its own SQLite database (`db-{branch}.sqlite3`). Export `GIT_BRANCH` for automatic isolation: +```bash +export GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD | sed 's/[^a-zA-Z0-9._-]/-/g' | tr '[:upper:]' '[:lower:]') +``` + 1. Fork the repository into your own personal GitHub account 2. Clone your fork: `git clone git@github.com:YourGitHubName/website.git` 3. Ensure you are running Python 3.13: `python -V` should output `Python 3.13.x` @@ -132,6 +203,12 @@ task heroku:releases # Show deployment history task heroku:rollback # Rollback to previous release task heroku:maintenance:on # Enable maintenance mode task heroku:maintenance:off # Disable maintenance mode + +# Branch Isolation (Docker) +task branch:info # Show current branch and environment info +task branch:verify # Verify database configuration +task branch:volumes # List all Docker volumes (all branches) +task branch:clean # Remove volumes for current branch (CAUTION!) ``` ### Direct Django Commands @@ -255,6 +332,12 @@ This project uses several tools to streamline development: - Rebuild Docker image: `task docker:build` - Reinstall dependencies: `uv sync --all-groups` +### Wrong Database Being Used +- Check current branch: `git branch` +- Verify environment: `task branch:info` +- Check database configuration: `task branch:verify` +- Ensure .env file is up to date: `task env:write` + ## Contributing 1. Fork the repository into your own GitHub account diff --git a/Taskfile.yaml b/Taskfile.yaml index e4e72ad..1510228 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -305,3 +305,16 @@ tasks: - docker volume rm pythonie-postgres-data-{{.GIT_BRANCH}} 2>/dev/null || echo "Postgres volume not found" - docker volume rm pythonie-redis-data-{{.GIT_BRANCH}} 2>/dev/null || echo "Redis volume not found" - echo "Volumes cleaned for branch {{.GIT_BRANCH}}" + + branch:verify: + desc: Verify database configuration for current branch + cmds: + - echo "=== Branch Configuration ===" + - echo "Git branch → {{.GIT_BRANCH}}" + - echo "Expected database → pythonie_{{.GIT_BRANCH}}" + - echo "" + - echo "=== Docker Environment ===" + - cat .env 2>/dev/null || echo ".env file not found (run any docker compose task to generate)" + - echo "" + - echo "=== PostgreSQL Databases ===" + - docker exec pythonie-postgres-{{.GIT_BRANCH}} psql -U postgres -c "\l" 2>/dev/null | grep pythonie || echo "PostgreSQL container not running" diff --git a/pythonie/pythonie/settings/dev.py b/pythonie/pythonie/settings/dev.py index 9e09010..800a24b 100644 --- a/pythonie/pythonie/settings/dev.py +++ b/pythonie/pythonie/settings/dev.py @@ -16,10 +16,12 @@ WAGTAILADMIN_BASE_URL = "http://localhost:8000" # SQLite (simplest install) +# Database name includes git branch for isolation between branches +GIT_BRANCH = os.environ.get("GIT_BRANCH", "dev") DATABASES = { "default": { "ENGINE": "django.db.backends.sqlite3", - "NAME": join(PROJECT_ROOT, "db.sqlite3"), + "NAME": join(PROJECT_ROOT, f"db-{GIT_BRANCH}.sqlite3"), } } From ef7051e06cecea3a13c102a1bfb0a213a75d0b77 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Fri, 25 Sep 2026 08:01:10 +0200 Subject: [PATCH 3/5] refactor(docker): rely on worktrunk hooks for per-worktree isolation Replace the home-grown git branch detection with worktrunk lifecycle hooks declared in .config/wt.toml: - pre-start writes a .env with COMPOSE_PROJECT_NAME, a per-branch DOCKER_IMAGE tag and hash_port-derived WEB_PORT / PGPORT - post-start runs `uv sync --all-groups` for the host virtualenv - post-remove tears down the worktree's compose project (containers, network, volumes) and its image docker-compose.yml reads these variables with the previous values as defaults, so nothing changes without worktrunk. The Taskfile loads .env (for PGPORT on the host) and builds through compose so the image tag comes from a single place. Drop the branch:* tasks, env:write, GIT_BRANCH plumbing, per-branch SQLite (the file already lives inside each worktree), the Redis 7 and healthcheck changes (unrelated), and the unused minio/mc services whose fixed host ports would collide between worktrees. --- .config/wt.toml | 49 +++++++ .env.example | 16 --- .gitignore | 4 +- DOCKER-BRANCHES.md | 224 ------------------------------ README.md | 85 +----------- Taskfile.yaml | 88 ++---------- docker-compose.yml | 55 ++------ pythonie/pythonie/settings/dev.py | 4 +- 8 files changed, 74 insertions(+), 451 deletions(-) create mode 100644 .config/wt.toml delete mode 100644 .env.example delete mode 100644 DOCKER-BRANCHES.md diff --git a/.config/wt.toml b/.config/wt.toml new file mode 100644 index 0000000..c1b1c74 --- /dev/null +++ b/.config/wt.toml @@ -0,0 +1,49 @@ +# Worktrunk project configuration (https://worktrunk.dev). +# +# Gives every worktree its own Docker Compose environment so several branches +# can run `task run` / `task tests` side by side. docker-compose.yml and the +# Taskfile read the `.env` written below and fall back to the historical +# defaults (port 8000, image python.ie/website-dev) when it is absent, so +# nothing changes for contributors who do not use worktrunk. +# +# The naming templates in pre-start and post-remove must stay in sync. + +# Blocking, so the `.env` exists before anyone runs docker compose in the new +# worktree. It pins, per branch: +# - COMPOSE_PROJECT_NAME: namespaces containers, networks and volumes +# (including the Postgres data volume), so each worktree has its own database; +# - DOCKER_IMAGE: a per-branch tag, so rebuilding the image on a branch that +# changes dependencies does not break the other worktrees; +# - WEB_PORT / PGPORT: deterministic host ports (hash_port), so published ports +# never collide. PGPORT is also what psql / `task database:pull` use on the host. +[pre-start] +env = """ +cat > .env </dev/null || true +""" + +# Shows each worktree's dev server URL in `wt list` (matches WEB_PORT above). +[list] +url = "http://localhost:{{ branch | hash_port }}" diff --git a/.env.example b/.env.example deleted file mode 100644 index a0635f1..0000000 --- a/.env.example +++ /dev/null @@ -1,16 +0,0 @@ -# Docker Compose Environment Variables -# Copy this file to .env and customize as needed - -# Override ports for parallel development (optional) -# Useful when running multiple branches simultaneously -# WEB_PORT=8001 -# PG_PORT=5433 - -# Override git branch detection (optional - normally auto-detected) -# GIT_BRANCH=my-custom-branch - -# Override Docker image name (optional) -# DOCKER_IMAGE=python.ie/website:custom-tag - -# Override database name (optional) -# PGDATABASE=pythonie_custom diff --git a/.gitignore b/.gitignore index a86116c..b954306 100644 --- a/.gitignore +++ b/.gitignore @@ -66,14 +66,14 @@ project.db # Environment files with sensitive credentials .envrc -.env development.env production.env +# Per-worktree Docker Compose overrides (generated, see .config/wt.toml) +.env # Database dumps and local databases *.dump *.duckdb -*.sqlite3 # Local settings pythonie/pythonie/settings/pgdev.py diff --git a/DOCKER-BRANCHES.md b/DOCKER-BRANCHES.md deleted file mode 100644 index cef9158..0000000 --- a/DOCKER-BRANCHES.md +++ /dev/null @@ -1,224 +0,0 @@ -# Docker Multi-Branch Development - -This setup allows you to run isolated Docker environments for each git branch, preventing conflicts when switching branches or working on multiple features simultaneously. - -## How It Works - -### Automatic Branch Detection - -The system automatically: -- Detects your current git branch (`git rev-parse --abbrev-ref HEAD`) -- Normalizes the branch name (e.g., `feature/add-auth` → `feature-add-auth`) -- Uses this normalized name for: - - Docker image tags: `python.ie/website:feature-add-auth` - - Database names: `pythonie_feature-add-auth` - - Container names: `pythonie-web-feature-add-auth` - - Docker volumes: `postgres-data-feature-add-auth` - -### Branch Isolation - -Each branch gets: -- ✅ Its own Docker image -- ✅ Its own PostgreSQL database (isolated data) -- ✅ Its own SQLite database for local dev: `pythonie/db-{branch}.sqlite3` -- ✅ Its own Redis instance (isolated cache) -- ✅ Its own Docker volumes (persistent storage) -- ✅ Unique container names (no conflicts) - -## Usage - -### Basic Commands - -```bash -# Check your current branch environment -task branch:info - -# Build image for current branch -task docker:build - -# Run development server (uses current branch) -task run - -# Run tests (uses current branch database) -task tests - -# Django shell for current branch -task django:shell-plus -``` - -### Working on Multiple Branches - -**Scenario**: You want to work on `feature/auth` while keeping `main` running. - -```bash -# Terminal 1: Main branch -git checkout main -task run -# → Uses: python.ie/website:main, pythonie_main database, port 8000 - -# Terminal 2: Feature branch (in another terminal) -git checkout feature/auth -task run -# → ERROR: Port 8000 already in use! -``` - -**Solution**: Use custom ports for parallel instances: - -```bash -# Terminal 1: Main branch -git checkout main -task run - -# Terminal 2: Feature branch with custom port -git checkout feature/auth -WEB_PORT=8001 PG_PORT=5433 task run -# → Uses: port 8001, separate database -``` - -### Managing Volumes - -```bash -# List all volumes (all branches) -task branch:volumes - -# Example output: -# pythonie-postgres-data-master -# pythonie-postgres-data-feature-add-auth -# pythonie-redis-data-master -# pythonie-redis-data-feature-add-auth - -# Clean volumes for current branch (DESTRUCTIVE!) -task branch:clean -# → Deletes all data for current branch -``` - -### Switching Branches - -When you switch branches, the system automatically uses the correct environment: - -```bash -# On main branch -git checkout main -task run # Uses main database - -# Switch to feature branch -git checkout feature/add-auth -task run # Automatically uses feature-add-auth database -``` - -**Important**: You must rebuild the image after switching branches if dependencies changed: - -```bash -git checkout feature/new-dependencies -task docker:build # Rebuild image with new dependencies -task run -``` - -## Advanced: Custom Ports for Parallel Development - -Create a `.env` file (git-ignored) to override default ports: - -```bash -# .env -WEB_PORT=8001 -PG_PORT=5433 -``` - -Or pass them inline: - -```bash -WEB_PORT=8001 PG_PORT=5433 task run -``` - -## Troubleshooting - -### Port Already in Use - -``` -Error: port 8000 already in use -``` - -**Solution**: Stop other instances or use custom ports: -```bash -task down # Stop all containers for current branch -# OR -WEB_PORT=8001 PG_PORT=5433 task run -``` - -### Database Not Found - -``` -FATAL: database "pythonie_feature-xyz" does not exist -``` - -**Solution**: Run migrations to create the database: -```bash -task django:migrate -``` - -### Wrong Database Being Used - -```bash -# Check which environment is active -task branch:info - -# Ensure you're on the correct git branch -git branch - -# Rebuild if needed -task docker:build -``` - -### Cleaning Up Old Branches - -After deleting a git branch, clean up its Docker resources: - -```bash -# Remove unused images -docker image prune -a - -# Remove volumes for deleted branches -# (replace 'old-branch-name' with actual normalized branch name) -docker volume rm pythonie-postgres-data-old-branch-name -docker volume rm pythonie-redis-data-old-branch-name - -# Or use the helper (must be on that branch) -git checkout old-branch-name -task branch:clean -``` - -## Environment Variables Reference - -| Variable | Default | Description | -|----------|---------|-------------| -| `GIT_BRANCH` | Auto-detected | Current git branch (normalized) | -| `DOCKER_IMAGE` | `python.ie/website:{branch}` | Docker image name | -| `PGDATABASE` | `pythonie_{branch}` | PostgreSQL database name | -| `WEB_PORT` | `8000` | Web server port | -| `PG_PORT` | `5432` | PostgreSQL port | - -## Tips - -1. **Branch Naming**: Use descriptive branch names. They'll appear in container names and logs. - -2. **Disk Space**: Each branch creates separate volumes. Monitor disk usage: - ```bash - docker system df - ``` - -3. **Production**: This system is for **development only**. Production uses fixed names and ports. - -4. **Database Migrations**: Each branch has independent migrations. Remember to run `task django:migrate` after switching branches with schema changes. - -5. **Shared Data**: If you need to copy data between branches: - ```bash - # Export from main - git checkout main - task run:postgres # Start just postgres - pg_dump -U postgres pythonie_main > /tmp/main.sql - - # Import to feature branch - git checkout feature/xyz - task run:postgres - psql -U postgres pythonie_feature-xyz < /tmp/main.sql - ``` diff --git a/README.md b/README.md index a3f2d45..ba51e72 100644 --- a/README.md +++ b/README.md @@ -12,8 +12,6 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 ## Quick Start (Docker - Recommended) -**Note**: Each git branch automatically gets its own isolated database and Docker environment. See [Docker Branch Isolation](#docker-branch-isolation) below. - 1. Build the Docker image: ```bash task docker:build @@ -22,7 +20,7 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 2. Start supporting services: ```bash - docker compose up -d postgres redis + docker compose up -d postgres redis minio ``` 3. Run database migrations: @@ -50,79 +48,10 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 7. Visit http://127.0.0.1:8000/ to see the site with sample content 8. Access Wagtail admin at http://127.0.0.1:8000/admin/ -## Docker Branch Isolation - -The project automatically isolates Docker environments by git branch, allowing you to work on multiple branches simultaneously without conflicts. - -### How It Works - -Each git branch gets: -- **Separate Docker image**: `python.ie/website:{branch-name}` -- **Isolated PostgreSQL database**: `pythonie_{branch-name}` (for docker-compose with pgdev settings) -- **Isolated SQLite database**: `pythonie/db-{branch-name}.sqlite3` (for local dev) -- **Dedicated volumes**: `pythonie-postgres-data-{branch}`, `pythonie-redis-data-{branch}` -- **Unique containers**: `pythonie-web-{branch}`, `pythonie-postgres-{branch}`, `pythonie-redis-{branch}` - -### Branch Commands - -```bash -# Check your current branch environment -task branch:info - -# Verify database configuration -task branch:verify - -# List all Docker volumes (all branches) -task branch:volumes - -# Clean up volumes for current branch (DESTRUCTIVE!) -task branch:clean -``` - -### Working on Multiple Branches - -**Scenario**: Work on `main` and `feature/new-auth` simultaneously. - -```bash -# Terminal 1: Main branch on default ports -git checkout main -task run # Uses pythonie_main database on port 8000 - -# Terminal 2: Feature branch on custom ports -git checkout feature/new-auth -WEB_PORT=8001 PG_PORT=5433 task run # Uses pythonie_feature-new-auth on port 8001 -``` - -### Switching Branches - -When you switch branches, the system automatically uses the correct environment: - -```bash -git checkout main -task run # Automatically uses main database - -git checkout feature/xyz -task run # Automatically uses feature-xyz database (isolated from main) -``` - -**Note**: Rebuild the Docker image after switching branches if dependencies changed: -```bash -git checkout feature/new-dependencies -task docker:build -task run -``` - -For detailed documentation, see [DOCKER-BRANCHES.md](DOCKER-BRANCHES.md). - ## Local Setup (Without Docker) If you prefer to develop without Docker: -**Note**: When developing locally, each git branch automatically uses its own SQLite database (`db-{branch}.sqlite3`). Export `GIT_BRANCH` for automatic isolation: -```bash -export GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD | sed 's/[^a-zA-Z0-9._-]/-/g' | tr '[:upper:]' '[:lower:]') -``` - 1. Fork the repository into your own personal GitHub account 2. Clone your fork: `git clone git@github.com:YourGitHubName/website.git` 3. Ensure you are running Python 3.13: `python -V` should output `Python 3.13.x` @@ -203,12 +132,6 @@ task heroku:releases # Show deployment history task heroku:rollback # Rollback to previous release task heroku:maintenance:on # Enable maintenance mode task heroku:maintenance:off # Disable maintenance mode - -# Branch Isolation (Docker) -task branch:info # Show current branch and environment info -task branch:verify # Verify database configuration -task branch:volumes # List all Docker volumes (all branches) -task branch:clean # Remove volumes for current branch (CAUTION!) ``` ### Direct Django Commands @@ -332,12 +255,6 @@ This project uses several tools to streamline development: - Rebuild Docker image: `task docker:build` - Reinstall dependencies: `uv sync --all-groups` -### Wrong Database Being Used -- Check current branch: `git branch` -- Verify environment: `task branch:info` -- Check database configuration: `task branch:verify` -- Ensure .env file is up to date: `task env:write` - ## Contributing 1. Fork the repository into your own GitHub account diff --git a/Taskfile.yaml b/Taskfile.yaml index 1510228..2d90767 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -3,28 +3,17 @@ version: '3' -vars: - # Detect current git branch and normalize the name - # (replace / with - and remove special characters) - GIT_BRANCH: - sh: git rev-parse --abbrev-ref HEAD 2>/dev/null | sed 's/[^a-zA-Z0-9._-]/-/g' | tr '[:upper:]' '[:lower:]' || echo "dev" - env: - # Pass branch name to docker-compose - GIT_BRANCH: "{{.GIT_BRANCH}}" - # Docker image tagged by branch - DOCKER_IMAGE: "python.ie/website:{{.GIT_BRANCH}}" - # Database isolated by branch - PGDATABASE: "pythonie_{{.GIT_BRANCH}}" + PGDATABASE: pythonie PGPASSWORD: pythonie PGUSER: postgres PGHOST: 127.0.0.1 HEROKU_APP: pythonie - # Dynamic ports (optional - can stay fixed if running only one instance at a time) - WEB_PORT: 8000 - PG_PORT: 5432 -dotenv: ['production.env'] +# `.env` holds per-worktree overrides (COMPOSE_PROJECT_NAME, DOCKER_IMAGE, +# WEB_PORT, PGPORT) written by worktrunk, see .config/wt.toml. Missing files +# are ignored, so everything falls back to the defaults in docker-compose.yml. +dotenv: ['.env', 'production.env'] tasks: database:drop: @@ -113,68 +102,48 @@ tasks: cmds: - heroku maintenance:off - env:write: - desc: Write environment variables to .env file for docker-compose - cmds: - - | - cat > .env </dev/null || echo "Postgres volume not found" - - docker volume rm pythonie-redis-data-{{.GIT_BRANCH}} 2>/dev/null || echo "Redis volume not found" - - echo "Volumes cleaned for branch {{.GIT_BRANCH}}" - - branch:verify: - desc: Verify database configuration for current branch - cmds: - - echo "=== Branch Configuration ===" - - echo "Git branch → {{.GIT_BRANCH}}" - - echo "Expected database → pythonie_{{.GIT_BRANCH}}" - - echo "" - - echo "=== Docker Environment ===" - - cat .env 2>/dev/null || echo ".env file not found (run any docker compose task to generate)" - - echo "" - - echo "=== PostgreSQL Databases ===" - - docker exec pythonie-postgres-{{.GIT_BRANCH}} psql -U postgres -c "\l" 2>/dev/null | grep pythonie || echo "PostgreSQL container not running" diff --git a/docker-compose.yml b/docker-compose.yml index d3c282d..676973c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,72 +1,43 @@ services: web: build: . - image: "python.ie/website:${GIT_BRANCH:-dev}" - container_name: "pythonie-web-${GIT_BRANCH:-dev}" + # Image, host ports and the compose project name can be overridden from a + # `.env` file next to this one (worktrunk writes it for every worktree, see + # .config/wt.toml). The defaults below apply when there is no `.env`. + image: ${DOCKER_IMAGE:-python.ie/website-dev} ports: - "${WEB_PORT:-8000}:8000" command: /usr/bin/fish depends_on: - postgres: - condition: service_healthy - redis: - condition: service_healthy + - postgres + - redis env_file: - development.env environment: - PGDATABASE: "pythonie_${GIT_BRANCH:-dev}" +# DJANGO_SETTINGS_MODULE: pythonie.settings.dev + PGDATABASE: pythonie PGUSER: postgres PGPASSWORD: pythonie - GIT_BRANCH: "${GIT_BRANCH:-dev}" volumes: - .:/app working_dir: /app + postgres: image: postgres:17 - container_name: "pythonie-postgres-${GIT_BRANCH:-dev}" - # ports: - # - "${PG_PORT:-5432}:5432" + ports: + - "${PGPORT:-5432}:5432" environment: POSTGRES_PASSWORD: pythonie - POSTGRES_DB: "pythonie_${GIT_BRANCH:-dev}" - healthcheck: - test: ["CMD-SHELL", "pg_isready -U postgres"] - interval: 10s - timeout: 5s - retries: 5 - restart: unless-stopped - volumes: - - postgres-data:/var/lib/postgresql/data redis: - restart: unless-stopped - image: redis:7-alpine - container_name: "pythonie-redis-${GIT_BRANCH:-dev}" - volumes: - - redis-data:/data - command: redis-server --appendonly yes - healthcheck: - test: ["CMD", "redis-cli", "ping"] - interval: 10s - timeout: 3s - retries: 3 + image: redis:6.2 test: - image: "python.ie/website:${GIT_BRANCH:-dev}" + image: ${DOCKER_IMAGE:-python.ie/website-dev} environment: DJANGO_SETTINGS_MODULE: pythonie.settings.tests - PGDATABASE: "pythonie_${GIT_BRANCH:-dev}" - depends_on: - postgres: - condition: service_healthy volumes: - .:/app working_dir: /app command: python pythonie/manage.py test pythonie --verbosity=3 - -volumes: - postgres-data: - name: "pythonie-postgres-data-${GIT_BRANCH:-dev}" - redis-data: - name: "pythonie-redis-data-${GIT_BRANCH:-dev}" diff --git a/pythonie/pythonie/settings/dev.py b/pythonie/pythonie/settings/dev.py index 800a24b..9e09010 100644 --- a/pythonie/pythonie/settings/dev.py +++ b/pythonie/pythonie/settings/dev.py @@ -16,12 +16,10 @@ WAGTAILADMIN_BASE_URL = "http://localhost:8000" # SQLite (simplest install) -# Database name includes git branch for isolation between branches -GIT_BRANCH = os.environ.get("GIT_BRANCH", "dev") DATABASES = { "default": { "ENGINE": "django.db.backends.sqlite3", - "NAME": join(PROJECT_ROOT, f"db-{GIT_BRANCH}.sqlite3"), + "NAME": join(PROJECT_ROOT, "db.sqlite3"), } } From f768770f30264c043d56f871ae112f965e76a103 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Fri, 25 Sep 2026 08:01:54 +0200 Subject: [PATCH 4/5] docs: document the worktrunk per-worktree Docker workflow Add a "Parallel Worktrees with worktrunk" section to DEVELOPMENT.md, link it from the README quick start, and drop the removed minio service from the docs. --- CLAUDE.md | 2 +- DEVELOPMENT.md | 36 ++++++++++++++++++++++++++++++++++++ README.md | 4 +++- 3 files changed, 40 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7d13f9c..ea88654 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -68,7 +68,7 @@ uv run python pythonie/manage.py runserver --settings=pythonie.settings.dev ### Docker Development (preferred) -Uses Task for most operations. Requires docker-compose with services: web, postgres, redis, minio. +Uses Task for most operations. Requires docker-compose with services: web, postgres, redis. With worktrunk (`wt`), `.config/wt.toml` writes a per-worktree `.env` (compose project name, image tag, ports) and cleans up the worktree's containers and volumes on `wt remove`; see DEVELOPMENT.md. ```bash # Build docker image diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 73007fd..d199207 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -147,6 +147,42 @@ uv run python pythonie/manage.py createsuperuser --settings=pythonie.settings.de uv run python pythonie/manage.py runserver --settings=pythonie.settings.dev ``` +### Parallel Worktrees with worktrunk (optional) + +[worktrunk](https://worktrunk.dev) (`wt`) manages one git worktree per branch. The project hooks in `.config/wt.toml` give every worktree its own Docker Compose environment, so several branches can run `task run` or `task tests` at the same time without sharing a database, an image or a port. + +```bash +# Create a worktree for a new branch (runs the hooks below) +wt switch --create my-feature + +# Inside it, the usual commands work unchanged +task docker:build +task django:migrate +task run + +# Show every worktree with its dev server URL +wt list + +# Remove the worktree AND its Docker resources (see below) +wt remove my-feature +``` + +The first time, worktrunk asks you to approve the project hooks. Review them and approve with `wt config approvals add`. + +What the hooks do: + +| Hook | What it does | +|------|--------------| +| `pre-start` | Writes a `.env` in the new worktree with `COMPOSE_PROJECT_NAME`, `DOCKER_IMAGE`, `WEB_PORT` and `PGPORT`, all derived from the branch name | +| `post-start` | Runs `uv sync --all-groups` in the background to create the worktree's `.venv` | +| `post-remove` | Runs `docker compose -p down --volumes --remove-orphans` and deletes the branch image | + +Docker Compose reads `.env` automatically, so each worktree gets its own containers, network and volumes (including the PostgreSQL data), its own image tag (`python.ie/website-dev:`) and deterministic host ports. Check them with `cat .env`; the web server is at `http://localhost:$WEB_PORT/`. The Taskfile also loads `.env`, so host tools such as `psql` or `task database:pull` talk to the worktree's PostgreSQL on `PGPORT`. + +**Removing a worktree deletes its database.** `wt remove` (and `wt merge`, which removes the worktree) destroys the Docker volumes of that worktree's compose project. Other worktrees and the main checkout use different project names and are not affected. Dump anything you want to keep first. + +Without worktrunk, or in a worktree created before these hooks existed, there is no `.env` and the defaults apply: port 8000 for the web server, 5432 for PostgreSQL, image `python.ie/website-dev`, and a compose project named after the directory. To add the per-worktree `.env` to an existing worktree, run `wt hook pre-start` inside it. + ### Environment Variables Create `development.env` at the root (copy from `development.env.example` if it exists): diff --git a/README.md b/README.md index ba51e72..e422982 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 2. Start supporting services: ```bash - docker compose up -d postgres redis minio + docker compose up -d postgres redis ``` 3. Run database migrations: @@ -48,6 +48,8 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 7. Visit http://127.0.0.1:8000/ to see the site with sample content 8. Access Wagtail admin at http://127.0.0.1:8000/admin/ +Working on several branches at once? See [Parallel Worktrees with worktrunk](DEVELOPMENT.md#parallel-worktrees-with-worktrunk-optional) to give each worktree its own database, image and ports. + ## Local Setup (Without Docker) If you prefer to develop without Docker: From ef5808cd02e6a306b98a6ff1957d2c15f90a3973 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Fri, 25 Sep 2026 08:07:22 +0200 Subject: [PATCH 5/5] fix(wt): remove running one-off containers before compose down docker compose down skips running containers started with docker compose run (such as an open task run), which kept the network and the branch image in use after wt remove. --- .config/wt.toml | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/.config/wt.toml b/.config/wt.toml index c1b1c74..cd9b9d1 100644 --- a/.config/wt.toml +++ b/.config/wt.toml @@ -35,11 +35,16 @@ deps = "uv sync --all-groups" # Background, after the worktree is gone (runs from the primary worktree). # DESTRUCTIVE: removes the containers, network and volumes (Postgres and Redis -# data) of this worktree's compose project, then its per-branch image. `-p` -# restricts it to that single project; other worktrees and the primary -# checkout use different project names and are never touched. +# data) of this worktree's compose project, then its per-branch image. The +# label filter and `-p` restrict it to that single project; other worktrees +# and the primary checkout use different project names and are never touched. +# The first step is needed because `down` skips running one-off containers +# started by `docker compose run` (e.g. a `task run` left open), which would +# keep the network and the image in use. [post-remove] docker = """ +ids=$(docker ps -aq --filter label=com.docker.compose.project=pythonie-{{ branch | sanitize_db }}) +[ -z "$ids" ] || docker rm -f $ids docker compose -p pythonie-{{ branch | sanitize_db }} down --volumes --remove-orphans docker image rm python.ie/website-dev:{{ branch | sanitize_db }} 2>/dev/null || true """