diff --git a/docs/MAKEFILE.md b/docs/MAKEFILE.md new file mode 100644 index 0000000..a3c4165 --- /dev/null +++ b/docs/MAKEFILE.md @@ -0,0 +1,330 @@ +# Makefile Reference + +Complete guide to all `make` commands available in the project. + +## Quick Reference + +```bash +# Run a command +make + +# Show available commands (if implemented) +make help +``` + +## Environment Management + +### Starting & Stopping + +| Command | Description | +| ------------------------------- | ---------------------------------------------------------------- | +| `make up` | Start all services for current `ENVIRONMENT` (from `.env.infra`) | +| `make up SERVICES=` | Start specific service(s) | +| `make down` | Stop all running services | +| `make down SERVICES=` | Stop specific service(s) | +| `make restart` | Restart all services | +| `make restart SERVICES=` | Restart specific service(s) | +| `make restart-db` | Restart database containers to apply new user configurations | +| `make ps` | List running containers with status | + +**Examples:** + +```bash +make up +make down + +make up SERVICES=user-service +make restart SERVICES="order-service payment-service" +make down SERVICES=api-gateway + +make restart-db +``` + +### Initialization + +| Command | Description | +| ----------------- | --------------------------------------------------------------------------------------- | +| `make init` | Initialize all config files from examples (`.env.infra`, `init-users.conf`, etc.) | +| `make init-local` | Initialize local development environment (generates `compose/docker-compose.local.yml`) | +| `make check-env` | Verify `.env.infra` exists | + +**First-time setup:** + +```bash +make init +# Edit .env.infra with your settings +make up +``` + +## Service Interaction + +### Exec into Containers + +| Command | Description | +| -------------------- | -------------------------------- | +| `make exec-postgres` | Open PostgreSQL shell (`psql`) | +| `make exec-mongo` | Open MongoDB shell (`mongosh`) | +| `make exec-redis` | Open Redis CLI | +| `make exec-redpanda` | Open Redpanda shell (`rpk`) | +| `make exec-keycloak` | Bash shell in Keycloak container | +| `make exec-minio` | MinIO client (`mc`) | +| `make exec-nginx` | Bash shell in Nginx container | + +**Example:** + +```bash +# Connect to Postgres and list databases +make exec-postgres +# Inside: \l +``` + +### Logs + +| Command | Description | +| -------------------------------- | ----------------------------------------- | +| `make logs` | View logs from all services (follow mode) | +| `make logs SERVICE=user-service` | View logs from specific service | + +## Database Management + +### Backups + +| Command | Description | +| ---------------------- | -------------------------------------------------------- | +| `make backup-postgres` | Create PostgreSQL backup (stored in `backups/postgres/`) | +| `make backup-mongo` | Create MongoDB backup (stored in `backups/mongodb/`) | + +**Backup files:** + +- Postgres: `backups/postgres/backup_YYYYMMDD_HHMMSS.sql` +- MongoDB: `backups/mongodb/backup_YYYYMMDD_HHMMSS/` + +## SSL Certificate Management + +| Command | Description | +| --------------------- | -------------------------------------------------- | +| `make ssl-cert-init` | Obtain initial SSL certificates from Let's Encrypt | +| `make ssl-cert-renew` | Manually renew SSL certificates | +| `make ssl-setup-cron` | Setup automatic renewal via systemd/cron | + +**Requirements:** + +- `ENVIRONMENT=stage` or `prod` in `.env.infra` +- Valid `SSL_EMAIL`, `API_DOMAIN_NAME`, `KEYCLOAK_DOMAIN_NAME` configured +- DNS records pointing to server +- Ports 80/443 accessible + +**Usage:** + +```bash +# First time setup (after DNS configured) +make ssl-cert-init + +# Setup automatic renewal +make ssl-setup-cron +``` + +## Cleanup + +| Command | Description | +| -------------------- | -------------------------------------------------------------------------------------- | +| `make clean` | Remove containers and networks (keeps volumes) | +| `make clean-volumes` | ⚠️ Remove ALL volumes (deletes all data) | +| `make clean-local` | Remove local development files (`compose/docker-compose.local.yml`, `.env.*` services) | +| `make prune` | Docker system prune (remove unused images/containers) | + +**⚠️ Warning:** `make clean-volumes` is destructive and will delete all database data! + +## Testing + +| Command | Description | +| --------------------------------------------- | ----------------------------------------------- | +| `make test` | Run all Ansible tests (Molecule + ansible-lint) | +| `make test-ansible` | Run Molecule tests for all roles | +| `make test-ansible-role ROLE=openmeal-deploy` | Test specific Ansible role | +| `make test-ansible-lint` | Run ansible-lint on playbooks/roles | +| `make test-ansible-syntax` | Check Ansible syntax | + +**Java tests:** + +```bash +# Run tests for specific service +./mvnw -pl services/user-service test + +# Run all tests +./mvnw test +``` + +## Utilities + +| Command | Description | +| --------------------- | ------------------------------------------------------------------ | +| `make check-services` | Run `scripts/check-services.sh` to verify health of all containers | +| `make check-yq` | Verify `yq` is installed (required for local dev) | + +## Environment-Specific Behavior + +The `ENVIRONMENT` variable in `.env.infra` controls which services start: + +### local-dev + +```bash +ENVIRONMENT=local-dev make up +``` + +**Starts:** Postgres, MongoDB, Redis, MinIO, microservices + +**Skips:** Keycloak, Redpanda, Nginx, Certbot, Monitoring + +**Use case:** Developer laptop with limited resources + +### shared-dev + +```bash +ENVIRONMENT=shared-dev make up +``` + +**Starts:** Keycloak, Redpanda, Nginx + +**Use case:** Shared VDS for team development + +### stage + +```bash +ENVIRONMENT=stage make up +``` + +**Starts:** Full stack (no MinIO, uses cloud S3) + +**Use case:** Pre-production testing + +### prod + +```bash +ENVIRONMENT=prod make up +``` + +**Starts:** Full stack + Prometheus/Grafana monitoring + +**Use case:** Production deployment + +## Advanced Usage + +### Custom Docker Compose Commands + +The Makefile constructs Docker Compose commands based on environment. You can run custom commands: + +```bash +# View constructed command +make ps + +# Run custom compose command +docker compose -f docker-compose.yml -f compose/infra.yml --profile local-dev +``` + +### Selective Service Management + +```bash +# Start only specific services +docker compose up -d user-service postgres redis + +# Restart single service +docker compose restart user-service + +# View logs from multiple services +docker compose logs -f user-service api-gateway +``` + +### Local Development with Code Changes + +```bash +# 1. Generate local compose file (builds from source) +make init-local + +# 2. Edit microservices.local to select which services to run +nano microservices.local +# Uncomment: user-service + +# 3. Build and start +make docker-build +make up +``` + +## Troubleshooting + +### Services won't start + +```bash +# Check what's running +make ps + +# Check health status +make check-services + +# View logs +make logs SERVICE=user-service + +# Clean restart +make down +make clean +make up +``` + +### Port conflicts + +```bash +# Check what's using ports +sudo lsof -i :8080 +sudo lsof -i :5432 + +# Stop conflicting services or change ports in .env.infra +``` + +### Database connection issues + +```bash +# Verify database is running +make exec-postgres + +# Check database users +make exec-postgres +# Inside: \du + +# Recreate database configs +make prepare-db-configs +make restart +``` + +### SSL certificate issues + +```bash +# Check certificate status +make exec-certbot +# Inside: certbot certificates + +# Force renewal +make ssl-cert-renew + +# Check Nginx config +make exec-nginx +# Inside: nginx -t +``` + +## Makefile Structure + +The main `Makefile` includes several modular files: + +- `makefiles/common.mk` - Common variables, colors, OS detection +- `makefiles/docker.mk` - Docker Compose orchestration +- `makefiles/services.mk` - Service exec commands and backups +- `makefiles/ssl.mk` - SSL certificate management +- `makefiles/ansible.mk` - Ansible testing +- `makefiles/local-dev.mk` - Local development utilities + +This modular approach keeps the Makefile maintainable and organized. + +## Related Documentation + +- [Architecture Overview](ARCHITECTURE.md) - System design and deployment model +- [Infrastructure Guide](../infrastructure/README.md) - Ansible deployment +- [Root README](../README.md) - Quick start guide diff --git a/infrastructure/README.md b/infrastructure/README.md new file mode 100644 index 0000000..2076465 --- /dev/null +++ b/infrastructure/README.md @@ -0,0 +1,480 @@ +# Infrastructure & Deployment Guide + +This directory contains Ansible automation for deploying the OpenMeal platform across environments. + +## Quick Start + +### Prerequisites + +**On Control Machine (CI/CD server or local):** + +- Ansible 2.15+ +- Python 3.13+ +- SSH access to target servers + +**On Target Servers:** + +- Ubuntu/Debian Linux +- SSH server running +- Sudo access for deployment user + +**For CI/CD:** + +- GitHub Secrets configured + +### Manual Deployment + +```bash +cd infrastructure/ansible + +# Deploy to staging +ansible-playbook -i inventories/stage playbooks/deploy.yml + +# Deploy to production +ansible-playbook -i inventories/prod playbooks/deploy.yml +``` + +### CI/CD Deployment + +**Important:** CI/CD only deploys application updates (Docker images). Infrastructure setup is done manually once. + +**Deployment Flow:** + +1. **One-time:** Manual infrastructure setup via `deploy.yml` +2. **Ongoing:** CI/CD runs `release.yml` to deploy new service versions + +**Automated via GitHub Actions:** + +- **Staging:** Auto-deploy on push to `main` branch +- **Production:** Manual trigger with Git tag + +See `.github/workflows/deploy-staging.yml` for pipeline details. + +## Directory Structure + +``` +infrastructure/ +├── ansible/ +│ ├── inventories/ +│ │ ├── stage/ +│ │ │ ├── hosts.yml +│ │ │ └── group_vars/ +│ │ │ ├── all.yml +│ │ │ └── vault.yml +│ │ └── prod/ +│ ├── playbooks/ +│ │ ├── deploy.yml +│ │ └── release.yml +│ └── roles/ +│ ├── common/ +│ ├── docker/ +│ ├── security/ +│ ├── openmeal-deploy/ +│ └── backup/ +└── keycloak/ + └── Dockerfile +``` + +**Key components:** + +- `inventories/` - Environment-specific configurations (stage, prod, shared-dev) +- `playbooks/` - deploy.yml (initial setup), release.yml (CI/CD updates) +- `roles/` - Modular Ansible roles for different deployment aspects +- `keycloak/` - Custom Keycloak Docker image + +## Playbooks + +### deploy.yml - Full Deployment + +Use for **initial server setup** or **complete redeployment**. + +**What it does:** + +1. Pre-flight checks (system resources, DNS, ports) +2. Install base packages and configure system +3. Install Docker and Docker Compose +4. Configure firewall and security hardening +5. Deploy application and configure services +6. Setup SSL certificates (stage/prod) +7. Configure automated backups (stage/prod) + +**Usage:** + +```bash +ansible-playbook -i inventories/stage/hosts.yml playbooks/deploy.yml + +# Dry run +ansible-playbook -i inventories/stage/hosts.yml playbooks/deploy.yml --check +``` + +**Note:** Role execution is controlled via `all.yml` variables, not `--skip-tags`. + +### release.yml - Incremental Release + +Use for **updating service versions** (used by CI/CD). + +**Git as Single Source of Truth:** + +- Ansible runs on CI/CD server with cloned repository +- Selectively syncs only necessary files to target server +- Source code never touches production (only Docker images) +- See [ADR-004](../docs/adr/004-git-as-single-source-of-truth.md) + +**What it does:** + +1. Sync compose files and configs from Git repo to target +2. Generate `.env.infra` with updated versions and secrets +3. Generate service-specific `.env` files +4. Pull updated Docker images from registry +5. Deploy services via `docker compose up -d` +6. Tag deployed images as `:latest` in registry +7. Cleanup old images (keep current versions) + +**Usage:** + +```bash +# Manual release (update versions in all.yml first) +ansible-playbook -i inventories/stage playbooks/release.yml + +# CI/CD release (with release_map) +ansible-playbook -i inventories/stage playbooks/release.yml \ + -e 'release_map={"user-service":"sha-abc123","api-gateway":"sha-def456"}' \ + -e 'registry_id=crXXXXXXXXXX' +``` + +## Inventory Configuration + +### hosts.yml + +Defines target servers: + +```yaml +all: + hosts: + openmeal-stage: + ansible_host: 192.168.1.100 + ansible_user: deploy + ansible_port: 22 +``` + +### group_vars/all.yml + +Environment-specific configuration: + +```yaml +deployment_type: stage +api_domain_name: api.example.com +keycloak_domain_name: auth.example.com +ssl_email: admin@example.com + +# Service versions (updated by CI/CD) +service_versions: + user_service: "sha-abc123" + api_gateway: "sha-def456" +``` + +**Note:** `deployment_type` can be `shared-dev`, `stage`, or `prod`. The `shared-dev` environment is deployed manually, not via CI/CD. + +### group_vars/vault.yml + +Encrypted secrets: + +```yaml +vault_postgres_password: "secret123" +vault_redis_password: "secret456" +vault_services_env: | + USER_SERVICE_DB_PASSWORD=xxx + USER_SERVICE_REDIS_URL=xxx +``` + +**Generation:** + +- **CI/CD:** Generated by `.github/scripts/generate-vault.py` from GitHub Secrets +- **Local:** Manually created for local Ansible runs +- **Note:** File is gitignored, never committed + +### CI/CD Pipeline + +GitHub Secrets → `generate-vault.py` → `vault.yml` → `.env` files + +**Required GitHub Secrets:** + +| Secret | Description | Format | +| ------------------------- | ------------------------------- | ------------------------------------ | +| `ANSIBLE_SSH_KEY` | SSH private key for deployment | PEM format | +| `ANSIBLE_INVENTORY_HOSTS` | Server inventory | YAML content of `hosts.yml` | +| `ANSIBLE_INVENTORY_ALL` | Environment config | YAML content of `all.yml` | +| `INFRA_ENV` | Infrastructure secrets | Multiline `KEY=VALUE` | +| `_ENV` | Per-service environment secrets | Multiline `KEY=VALUE` | +| `SERVICES_ENV` (optional) | Consolidated service secrets | Multiline `_KEY=VALUE` | +| `YC_REGISTRY_ID` | Yandex Container Registry ID | String | +| `YC_REGISTRY_USERNAME` | Registry username | String | +| `YC_REGISTRY_PASSWORD` | Registry password | String | + +**Example INFRA_ENV:** + +``` +POSTGRES_PASSWORD=secret123 +MONGODB_ROOT_PASSWORD=secret456 +REDIS_PASSWORD=secret789 +REDPANDA_SUPERUSER_PASSWORD=secretabc +``` + +**Per-Service Secrets (recommended):** + +Use `_ENV` secrets for each service: + +**Example USER_SERVICE_ENV:** + +``` +DB_PASSWORD=user_svc_secret +REDIS_URL=redis://redis:6379/0 +JWT_SECRET=user_jwt_key +``` + +**Example ORDER_SERVICE_ENV:** + +``` +DB_PASSWORD=order_svc_secret +KAFKA_BOOTSTRAP_SERVERS=redpanda:9092 +PAYMENT_API_KEY=payment_secret +``` + +**Consolidated Secrets (optional):** + +Alternatively, use `SERVICES_ENV` for shared configuration across multiple services: + +**Example SERVICES_ENV:** + +``` +USER_SERVICE_DB_PASSWORD=xxx +USER_SERVICE_REDIS_URL=redis://... +ORDER_SERVICE_DB_PASSWORD=yyy +ORDER_SERVICE_KAFKA_BOOTSTRAP_SERVERS=redpanda:9092 +``` + +Per-service secrets take priority over consolidated secrets when both are defined. + +### Variable Flow + +``` +GitHub Secrets + ↓ +generate-vault.py (Python script) + ↓ +vault.yml (Ansible variables) + ↓ +generate-configs.yml (Ansible task) + ↓ +.env.infra + .env.{service} (on server) + ↓ +Docker Compose (container environment) +``` + +## Roles + +### common + +- Create application user/group +- Install base packages (curl, git, etc.) +- Configure system settings + +### docker + +- Install Docker Engine +- Install Docker Compose v2 +- Configure Docker daemon +- Add app user to docker group + +### security + +- Configure UFW firewall (allow 22, 80, 443) +- Install and configure fail2ban +- Setup unattended-upgrades +- Disable root SSH login + +### openmeal-deploy + +- Sync project files to server (rsync) +- Generate `.env.infra` from vault +- Generate service `.env` files +- Configure Redpanda SASL/SCRAM +- Setup SSL certificates (via Certbot) +- Start services via Makefile + +### backup + +- Configure PostgreSQL backups (daily cron) +- Configure MongoDB backups (daily cron) +- Retention: 7 days local, 30 days remote (optional) + +## SSL Certificate Management + +SSL certificates are **automatically managed by Ansible** during deployment. + +### Automatic Setup & Renewal + +The `openmeal-deploy` role handles: + +- Initial certificate issuance via Let's Encrypt +- Automatic renewal (systemd timer or cron) +- Nginx reload after renewal +- Certificate validation and monitoring + +Certificates renew automatically when <30 days remaining. + +### Manual Operations (Optional) + +If needed, SSL can be managed directly on target server: + +```bash +# Initial certificate +make ssl-cert-init +# Manual renewal +make ssl-cert-renew +``` + +Or via Docker Compose: + +```bash +docker compose run --rm certbot renew +docker compose exec nginx nginx -s reload +``` + +## Backup & Restore + +### Automated Backups + +Configured by `backup` role (stage/prod only): + +- **PostgreSQL:** Daily at 2 AM +- **MongoDB:** Daily at 3 AM +- **Retention:** 7 days +- **Location:** `/var/backups/openmeal/` + +### Manual Backup + +```bash +# On target server +make backup-postgres +make backup-mongo +``` + +### Restore + +```bash +# PostgreSQL +docker compose exec -T postgres psql -U postgres < backup.sql + +# MongoDB +docker compose exec -T mongodb mongorestore --archive < backup.archive +``` + +## Troubleshooting + +### Deployment Fails at "Pull images" + +**Cause:** Docker not logged into registry + +**Fix:** + +```bash +# On target server (avoid password in shell history) +echo "$YC_REGISTRY_PASSWORD" | docker login cr.yandex --username oauth --password-stdin + +# Or ensure vault_yc_registry_username/password are set in Ansible +``` + +### Services Won't Start + +**Check logs:** + +```bash +# On target server +make logs SERVICE=user-service +make check-services +``` + +**Common issues:** + +- Port conflicts: Check `.env.infra` port assignments +- Missing secrets: Verify `vault.yml` was generated correctly +- Database not ready: Check healthchecks in `compose/infra.yml` + +### SSL Certificate Issues + +**Check certificate status:** + +```bash +make exec-certbot +# Inside: certbot certificates +``` + +**Common issues:** + +- DNS not pointing to server: Verify A records +- Port 80 blocked: Check firewall (`sudo ufw status`) +- Rate limit: Let's Encrypt has limits (5 certs/week per domain) + +### Ansible Connection Issues + +**Test connectivity:** + +```bash +ansible -i inventories/stage all -m ping +``` + +**Common issues:** + +- SSH key not added: `ssh-add ~/.ssh/deploy_key` +- Wrong user: Check `ansible_user` in `hosts.yml` +- Firewall blocking SSH: Ensure port 22 is open + +## Testing + +### Molecule Tests + +Test Ansible roles in isolated environments: + +```bash +# Test all roles +make test-ansible + +# Test specific role +make test-ansible-role ROLE=openmeal-deploy + +# Lint playbooks +make test-ansible-lint +``` + +### Syntax Check + +```bash +# Check playbook syntax +ansible-playbook playbooks/deploy.yml --syntax-check + +# Or use Makefile +make test-ansible-syntax +``` + +## Related Documentation + +- [Architecture Overview](../docs/ARCHITECTURE.md) - System design and deployment model +- [Makefile Reference](../docs/MAKEFILE.md) - Command reference +- [ADR-005: Ansible for Deployment Automation](../docs/adr/005-ansible.md) - Why we chose Ansible + +## CI/CD Pipeline + +See `.github/workflows/` for complete pipeline: + +1. **prepare-release** - Detect changed services, generate version +2. **build-images** - Build and push Docker images to registry +3. **deploy-ansible** - Run `release.yml` playbook with new versions + +**Flow:** + +``` +Git Push → Detect Changes → Build Images → Generate vault.yml → Ansible Deploy +``` + +For detailed CI/CD documentation, see [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md#build--deployment-pipeline).