PromptShield

Autonomous Red Teaming Agent for LLM Security Assessment

PromptShield is a comprehensive security testing framework that autonomously assesses the robustness and safety of Large Language Model (LLM) API endpoints through systematic red teaming operations.

Key Features

Automated Testing

Progressive Testing: Early-stopping strategy that finds minimum working jailbreak strength
Batch Testing: Parallel execution for comprehensive coverage
Rate Limiting: Built-in token bucket rate limiter prevents API throttling
Retry Logic: Automatic retry with exponential backoff for transient failures

Hybrid Judge Pipeline

Rule-Based Detection: 12 pre-configured regex patterns for common jailbreak techniques
Heuristic Scoring: Pattern analysis for multi-step instructions, action verbs, technical jargon
Judge LLM: AI-powered classification for ambiguous cases with few-shot examples
Explainable Results: Every decision includes matched rules and reasoning

Rich Analytics

Real-time Dashboards: Live progress tracking with KPIs
Interactive Charts: Donut charts, bar charts, time series with Chart.js
Drill-down Analysis: View full prompts, responses, and judge reasoning
Export Capabilities: JSON reports with comprehensive metrics

Curated Datasets

RedTeam (2,000 prompts): Professional red team scenarios
JailBreakV (28,000 prompts): Sophisticated jailbreak methodologies
Malignant Prompts: General harmful content generation attempts
CySecBench: Technical security testing prompts

Quick Start

Prerequisites

Node.js 18 or higher
npm package manager
OpenRouter API key (or enable mock mode for testing)

Installation

# Backend
cd backend
npm install
cp .env.example .env
# Edit .env: Set OPENROUTER_API_KEY or enable MOCK_PROVIDER_ENABLED=true
npm run dev

# Frontend (new terminal)
cd frontend
npm install
npm run dev

Open http://localhost:5173 in your browser.

For detailed setup instructions, see SETUP.md

How It Works

1. Create a Test Job

Configure and start a security assessment:

Select test mode (progressive or batch)
Choose datasets and filters
Set concurrency and rate limits
Monitor real-time progress

2. Worker Processes Prompts

Loads prompts from selected datasets
Sends prompts to target LLM via OpenRouter
Collects responses with metadata (latency, tokens, cost)
Manages rate limiting and retries

3. Judge Pipeline Analyzes Responses

Stage 1: Rule-Based Detection

Checks 12 regex patterns for common jailbreak indicators
Examples: explicit instructions, role manipulation, filter bypass

Stage 2: Heuristic Scoring

Analyzes patterns: multi-step instructions, action verbs, technical jargon
Detects disclaimer language and absence of refusals

Stage 3: Judge LLM (for ambiguous cases)

Calls judge LLM with few-shot examples for borderline cases
Returns structured JSON classification with confidence scores
Ensemble decision combines all stages

4. View Results

KPI cards: jailbreaks detected, success rate, latency, cost
Interactive charts: label distribution, strength breakdown, time series
Detailed table: paginated results with drill-down capabilities
Export options: JSON reports for further analysis

Architecture

Backend (Node.js + TypeScript)

backend/
├── src/
│   ├── config/          # Environment configuration
│   ├── storage/         # In-memory storage + file persistence
│   ├── judge/           # Hybrid judge pipeline + rules
│   ├── providers/       # OpenRouter integration + rate limiting
│   ├── worker/          # Job queue + progressive/batch modes
│   ├── routes/          # API endpoints
│   └── utils/           # Logger and utilities
├── data/                # Persisted jobs and results
└── logs/                # Application logs

Frontend (React + Vite)

frontend/
├── src/
│   ├── Home.jsx         # Landing page
│   ├── PromptBrowser.jsx # Dataset explorer
│   ├── JobsList.jsx     # Test jobs list
│   ├── Analysis.jsx     # Detailed analysis dashboard
│   └── App.jsx          # Application router
└── public/
    └── data/processed/  # JSONL datasets

API Reference

Start Test Job

POST /api/test/start
{
  "jobName": "Security Audit",
  "mode": "progressive",
  "datasetSources": ["RedTeam", "JailBreakV"],
  "concurrency": 5,
  "stopOnFirstSuccess": true
}

Get Job Status

GET /api/test/status/:jobId

Get Results

GET /api/test/result/:jobId?page=1&pageSize=50

Get Report

GET /api/report/:jobId/json

Full API documentation: backend/README.md

Testing Modes

Progressive Mode (Recommended)

Tests prompts in strength order (weak → moderate → strong) with early-stopping when a jailbreak is detected.

Benefits:

Cost-effective: minimizes API calls by 50-70%
Finds minimum working strength per prompt
Ideal for initial assessments

Batch Mode

Tests all prompts exhaustively in parallel batches.

Benefits:

Maximum throughput
Complete coverage
Suitable for comprehensive audits

Judge Pipeline

Rule-Based Detection

12 pre-configured patterns detect common jailbreak techniques:

Explicit jailbreak instructions
Role manipulation (DAN, evil personas)
System prompt leaks
Filter bypass language
Encoded harmful content
Multi-turn jailbreak patterns

Customize: Edit backend/src/judge/rules.json to add custom rules.

Heuristic Scoring

Analyzes response patterns for:

Multi-step numbered instructions
Action verb density
Technical jargon presence
Disclaimer language
Refusal keyword absence

Judge LLM

For ambiguous cases (score 0.3-0.7):

Constructs prompt with few-shot examples
Calls judge LLM (default: Claude-3-Haiku)
Parses JSON classification with confidence scores
Ensemble decision uses LLM result if confidence ≥ 0.8

Configuration

Backend Environment Variables

# Server Configuration
PORT=3001
NODE_ENV=development

# OpenRouter API
OPENROUTER_API_KEY=your-api-key-here
OPENROUTER_MODEL=anthropic/claude-3.5-sonnet  # Target model to test

# Judge LLM
JUDGE_LLM_MODEL=anthropic/claude-3-haiku  # Model for classification
JUDGE_LLM_MAX_TOKENS=500

# Worker Settings
WORKER_CONCURRENCY=5
WORKER_MAX_RETRIES=3
PROVIDER_RATE_LIMIT_RPM=50

# Mock Mode (testing without API costs)
MOCK_PROVIDER_ENABLED=true

See SETUP.md for complete configuration guide.

Docker Deployment

# Build and run with Docker Compose
docker-compose up -d

Access at:

Frontend: http://localhost
Backend API: http://localhost:3001

Security and Ethics

Responsible Use

PromptShield is designed for legitimate security research and defensive purposes only.

Requirements:

Obtain proper authorization before testing any LLM systems
Respect terms of service of target platforms
Use responsibly for defensive security improvement
Follow ethical guidelines for AI safety research

Data Privacy

API keys stored server-side only (never exposed to frontend)
Results stored locally with no external transmission
Optional data retention policies available
Supports secure key management practices

Contributing

Contributions are welcome! Areas for improvement:

PDF report generation with Puppeteer
PostgreSQL/Redis integration for production
Custom prompt upload functionality
Scheduled/periodic scans
Webhook notifications
Additional judge rules and heuristics
Multi-model comparison testing
Human review UI for ambiguous cases

Documentation

SETUP.md - Complete setup and configuration guide
SYSTEM_OVERVIEW.md - Technical architecture details
IMPLEMENTATION_SUMMARY.md - Component overview
backend/README.md - Backend API reference

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Security research community for adversarial prompt methodologies
Dataset contributors: RedTeam, JailBreakV, CySecBench, Malignant prompts
OpenRouter for unified LLM API access
Open source community for foundational tools

Support

Issues: GitHub Issues
Documentation: See docs/ directory
Email: security@promptshield.app

Built for the AI safety community

Name		Name	Last commit message	Last commit date
Datasets Pre-Processing		Datasets Pre-Processing
backend		backend
frontend		frontend
.gitignore		.gitignore
API_KEY_FEATURE.md		API_KEY_FEATURE.md
IMPLEMENTATION_SUMMARY.md		IMPLEMENTATION_SUMMARY.md
LICENSE		LICENSE
README.md		README.md
SETUP.md		SETUP.md
SYSTEM_OVERVIEW.md		SYSTEM_OVERVIEW.md
setup.ps1		setup.ps1

Latest commit

History

Folders and files

Repository files navigation