Proxy DB Engine is a CLI-first public proxy harvesting and validation project focused on building a reusable proxy database rather than just scraping raw lists. It combines harvesting, protocol-aware validation, handshake hardening, tunnel-aware downstream proof, scoring, export, and calibration into one single-node workflow.
Most public proxy projects stop at collecting raw lists or doing shallow liveness checks. This repo is trying to push one layer deeper: keep a local proxy database, validate protocol behavior more honestly, keep evidence history, and produce shortlists that other projects can actually consume.
The project is usable for research, batch validation, shortlist generation, and feeding downstream tooling. It has real harvesting, protocol-aware validation, downstream-proof tracking, export filters, and calibration support.
The honest real-world state right now is that public proxy quality is very poor. In a live 300-sample run, only a very small fraction survived the full evidence chain. That is useful information, not a failure of the tool.
- public proxy quality is extremely noisy in real-world runs
- the project is not yet a finished answer for highly sensitive targets
- there is no full TLS-native tunnel proof yet
- threshold defaults are still recommendation-driven, not auto-applied
- there is no API or dashboard layer yet
- harvest public proxy candidates from multiple sources
- normalize and deduplicate candidates
- protocol-aware validation with handshake-native hardening
- tunnel-aware downstream proof tracking
- scoring and shortlist generation
- export, stats, refresh, and calibration commands
- source adapters
- normalization and dedupe
- validation and protocol probes
- scoring and aggregate state
- SQLite-backed storage and history
- calibration and threshold sweep outputs
Flow summary:
flowchart TD
A[Public Sources] --> B[Normalization]
B --> C[Deduplication]
C --> D[Protocol Validation]
D --> E[Handshake-Native Proof]
E --> F[Tunnel-Aware Downstream Proof]
F --> G[Scoring]
G --> H[(SQLite State + History)]
H --> I[Export]
H --> J[Calibration]
J --> K[Threshold Sweep]
J --> L[Shortlists]
Detailed flow notes: docs/FLOWS.md
python3 -m venv .venv
. .venv/bin/activate
pip install -e .[dev]
proxydb harvest
proxydb validate --limit 300
proxydb export --format json --winning-protocol https --require-downstream-proof
proxydb calibrate --sample-limit 300proxydb harvestproxydb validate --limit Nproxydb export --format {text,json,csv}proxydb statsproxydb refresh --limit Nproxydb calibrate --sample-limit N
- harvest public proxy candidates
- validate a batch against the current evidence chain
- export or inspect surviving candidates
- run calibration to produce threshold sweep artifacts and shortlists
- review shortlist quality before adopting thresholds operationally
Calibration runs currently emit structured JSON artifacts for:
- summary metrics
- threshold sweep results
- overall shortlist
- per-protocol shortlists
Example sample artifact: assets/sample-calibration-summary.json
From the first real medium sample run:
- sample size: 300
- downstream-proof successes: 4
- winning protocol distribution: HTTPS only in that sample
- outcome: usable shortlist existed, but survival rate was very low
That is exactly why the repo includes calibration and threshold sweep support.
- research public proxy quality realistically
- build shortlist feeds for downstream automation
- compare protocol survival patterns
- tune filtering thresholds from live batch evidence instead of guesswork
- do not commit live database files, tokens, or sensitive runtime artifacts
- calibration artifacts should stay sanitized and summary-oriented when published
- sample output in this repo is intentionally limited to safe summary data
- deeper live validation and threshold tuning
- stronger per-protocol downstream proof
- source quality intelligence
- cleaner consumer-facing query surfaces
- later API or dashboard layer only after the evidence model settles