Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 128 additions & 28 deletions .agents/skills/rl/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,136 @@
# RL — Reinforcement Learning Experiments (Drush CLI)
---
name: rl
version: 1.0.0
description: >
Manage Reinforcement Learning (A/B testing) experiments via Drush CLI.
Thompson Sampling multi-armed bandit experiments: create, analyze,
configure, and optimize content variants.
triggers:
- /rl
- experiment
- a/b test
- ab test
- bandit
- thompson sampling
- conversion rate
- variant
---

Manage A/B testing experiments powered by Thompson Sampling via Drush.
# RL: Reinforcement Learning Experiments

## Commands
You are managing A/B testing experiments powered by Thompson Sampling.
The RL module tracks impressions (turns) and conversions (rewards) across
content variants and automatically optimizes traffic distribution.

### Discovery & Analysis
- `drush rl:list` — List all experiments with stats
- `drush rl:status <id>` — Experiment phase, confidence, distribution
- `drush rl:performance <id>` — Arm-level conversion rates and labels
- `drush rl:trends <id>` — Historical trends (--period=daily|weekly|monthly)
- `drush rl:export <id>` — Export full data (--snapshots for history)
- `drush rl:analyze <id>` — Full analysis with recommendations
## Preamble: Auto-discover Current State

### Experiment CRUD
- `drush rl:experiment:create <id> --module=X --name="Y"` — Create experiment
- `drush rl:experiment:update <id> --name="Y"` — Update metadata
- `drush rl:experiment:delete <id>` — Delete experiment + all data
```bash
# List all active experiments
drush rl:list --format=yaml

# Get current module configuration
drush rl:config:list

# If experiments exist, get status of the most active one
# drush rl:status <experiment_id> --format=yaml
```

## Commands Reference

### Discovery & Analysis (read-only)

| Command | Alias | Purpose |
|---|---|---|
| `rl:list` | `rll` | List all experiments with summary stats |
| `rl:status <id>` | `rlst` | Detailed status: phase, confidence, traffic distribution |
| `rl:performance <id>` | `rlp` | Arm-level performance with human-readable labels |
| `rl:trends <id>` | `rlt` | Historical trends with period aggregation |
| `rl:export <id>` | `rle` | Export complete experiment data |
| `rl:analyze <id>` | `rla` | Full analysis with actionable recommendations |

### Experiment Lifecycle

| Command | Alias | Purpose |
|---|---|---|
| `rl:experiment:create <id>` | `rl-ec` | Create experiment (`--module`, `--name`, `--dry-run`) |
| `rl:experiment:update <id>` | `rl-eu` | Update name/module (`--module`, `--name`, `--dry-run`) |
| `rl:experiment:delete <id>` | `rl-ed` | Delete experiment + all data (`--dry-run`) |

### Configuration
- `drush rl:config:list` — List all settings
- `drush rl:config:get <key>` — Get one setting
- `drush rl:config:set <key> <value>` — Set with validation
- `drush rl:config:reset` — Reset to defaults
- `drush rl:event-log:clear` — Clear all snapshots

### Settings Keys
| Key | Type | Default |

| Command | Alias | Purpose |
|---|---|---|
| `rl:config:get [key]` | `rl-cg` | Get one or all settings |
| `rl:config:set <key> <value>` | `rl-cs` | Set a setting with validation (`--dry-run`) |
| `rl:config:list` | `rl-cl` | List all settings with current values |
| `rl:config:reset` | `rl-cr` | Reset all settings to defaults (`--dry-run`) |
| `rl:event-log:clear` | `rl-elc` | Clear all snapshots (`--dry-run`) |

### Setup

| Command | Alias | Purpose |
|---|---|---|
| debug_mode | boolean | false |
| enable_event_log | boolean | true |
| event_log_max_rows | integer | 100000 |
| chart_line_threshold | integer | 9 |
| `rl:setup-ai` | `rl-sa` | Install AI skill files (`--host`, `--check`) |

## Configuration Keys

| Key | Type | Default | Description |
|---|---|---|---|
| `debug_mode` | boolean | false | Log Thompson Sampling scores |
| `enable_event_log` | boolean | true | Record snapshots for historical charts |
| `event_log_max_rows` | integer | 100000 | Max snapshot rows (1000–10000000) |
| `chart_line_threshold` | integer | 9 | Max variants for 2D line chart (2–50) |

## Workflow Examples

### Create and monitor a new A/B test
```bash
# Create experiment
drush rl:experiment:create hero_cta_test \
--module=my_module \
--name="Hero CTA Button Color Test"

# Check status after data collection
drush rl:status hero_cta_test

# Get detailed performance breakdown
drush rl:performance hero_cta_test --format=yaml

# Full analysis with recommendations
drush rl:analyze hero_cta_test
```

### Configure the module
```bash
# Enable debug logging temporarily
drush rl:config:set debug_mode true

# Increase snapshot storage
drush rl:config:set event_log_max_rows 500000

# Check all settings
drush rl:config:list
```

## Key Concepts

- **Experiment**: A multi-armed bandit test comparing content variants
- **Arm**: A single variant (identified by arm_id, often a node ID)
- **Turn/Impression**: One display of a variant to a visitor
- **Reward/Conversion**: A successful outcome (click, form submit, etc.)
- **Thompson Sampling**: Bayesian algorithm that balances
exploration vs exploitation
- **Phase**: exploration, learning, exploitation, conclusive
- **Confidence**: Statistical confidence that the top
performer is truly best

## Notes

All state-changing commands support `--dry-run`.
Analysis commands support `--format=json|yaml|table`.
- Experiments are tracked in custom database tables,
not Drupal config entities
- The `rl.php` endpoint provides high-performance
tracking with minimal Drupal bootstrap
- Arms are typically node IDs; the analyzer resolves
them to node titles automatically
- Views management is handled by `drush_webmaster`
(`wm:view:*` commands); do not duplicate
6 changes: 3 additions & 3 deletions .claude/skills/rl/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ triggers:
- variant
---

# RL — Reinforcement Learning Experiments
# RL: Reinforcement Learning Experiments

You are managing A/B testing experiments powered by Thompson Sampling.
The RL module tracks impressions (turns) and conversions (rewards) across
content variants and automatically optimizes traffic distribution.

## Preamble — Auto-discover Current State
## Preamble: Auto-discover Current State

```bash
# List all active experiments
Expand Down Expand Up @@ -133,4 +133,4 @@ drush rl:config:list
- Arms are typically node IDs; the analyzer resolves
them to node titles automatically
- Views management is handled by `drush_webmaster`
(`wm:view:*` commands) — do not duplicate
(`wm:view:*` commands); do not duplicate
62 changes: 62 additions & 0 deletions .github/workflows/skills-pair-check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
name: Skills pair check

on:
pull_request:
paths:
- '.claude/**'
- '.agents/**'
- '**/.claude/**'
- '**/.agents/**'

jobs:
pairs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Fail when paired skill files differ
run: |
fail=0
# Every SKILL.md whose skill exists in both trees must exist in
# both trees and be byte-identical. Skills living in only one
# tree (e.g. Claude-only helpers) are skipped by design.
while IFS= read -r claude_file; do
agents_file="${claude_file//\/.claude\//\/.agents\/}"
agents_dir=$(dirname "$agents_file")
if [ -d "$agents_dir" ]; then
if [ ! -f "$agents_file" ]; then
echo "MISSING: $agents_file (no .agents counterpart)"
fail=1
elif ! cmp -s "$claude_file" "$agents_file"; then
echo "DIFFER: $claude_file vs $agents_file"
fail=1
fi
fi
done < <(find . -type f -name 'SKILL.md' -path '*/.claude/*' \
-not -path './.git/*' -not -path '*/node_modules/*' \
-not -path '*/vendor/*' -not -path '*worktrees*')
while IFS= read -r agents_file; do
claude_file="${agents_file//\/.agents\//\/.claude\/}"
claude_dir=$(dirname "$claude_file")
if [ -d "$claude_dir" ] && [ ! -f "$claude_file" ]; then
echo "MISSING: $claude_file (no .claude counterpart)"
fail=1
fi
done < <(find . -type f -name 'SKILL.md' -path '*/.agents/*' \
-not -path './.git/*' -not -path '*/node_modules/*' \
-not -path '*/vendor/*' -not -path '*worktrees*')
# All other files are compared only when both counterparts
# exist; host-specific files such as agents/openai.yaml stay
# optional.
while IFS= read -r claude_file; do
agents_file="${claude_file//\/.claude\//\/.agents\/}"
if [ -f "$agents_file" ] \
&& ! cmp -s "$claude_file" "$agents_file"; then
echo "DIFFER: $claude_file vs $agents_file"
fail=1
fi
done < <(find . -type f -path '*/.claude/*' \
-not -name 'SKILL.md' -not -path './.git/*' \
-not -path '*/node_modules/*' -not -path '*/vendor/*' \
-not -path '*worktrees*')
exit $fail
Loading
Loading