Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
candidates against previously written places. Existing databases are
upgraded in place when appending; the new columns are nullable and readers
are unaffected.
- Add `sample` and `sweep` subcommands to `scripts/validate_with_locationiq.js`: a quota-aware, resumable world validation sweep against LocationIQ (JSONL response cache keyed by rounded coordinates, persisted per-UTC-day request cap, per-country Markdown mismatch report plus machine-readable mismatch JSONL, `--dry-run` for cache-only evaluation) and a GeoNames-TSV sampler that picks the top-N most populous places per country. Name comparison now preserves non-Latin scripts.
- Run the test suite in GitHub Actions on pushes to master and all pull
requests (Node 20.x and 22.x); raise Jasmine's per-hook timeout so
fixture-database construction survives slow shared runners
Expand Down
34 changes: 34 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,40 @@ It creates/updates:

Cache DB path is automatic (default behavior): `tmp/locationiq-validation-<database-basename>.sqlite`.

### World Validation Sweep (LocationIQ)

The same script also runs a quota-aware, resumable sweep over a world-wide
points file and ranks countries by mismatch rate, so data work can be aimed at
the worst areas first:

```bash
# 1. Build a points file from a GeoNames-style TSV (e.g. cities1000.txt):
# the top 25 most populous places per country. Not committed to the repo.
node scripts/validate_with_locationiq.js sample \
--geonames tmp/cities1000.txt --per-country 25

# 2. Run the sweep. Stays inside LocationIQ's free tier by default:
# max 4500 requests per UTC day (persisted across invocations) at 1 req/s.
LOCATIONIQ_API_KEY=... node scripts/validate_with_locationiq.js sweep \
--points tmp/locationiq-sweep/points.jsonl --database tmp/world.sqlite

# 3. Rebuild the report from cache only, no network:
node scripts/validate_with_locationiq.js sweep \
--points tmp/locationiq-sweep/points.jsonl --database tmp/world.sqlite --dry-run
```

Every LocationIQ response is cached as JSONL keyed by coordinates rounded to
four decimals, so re-running the same command never re-queries a cached point —
if a run stops at the daily cap or on HTTP 429, just run it again later to
resume. The daily-cap state lives outside the workdir (default
`tmp/locationiq-quota.json`, suffixed with a non-reversible fingerprint of the
API key), so every sweep configuration using one key shares a single cap, while
separate keys — which LocationIQ meters separately — keep their own tallies.
Outputs land under `--workdir` (default `tmp/locationiq-sweep/`):
`report.md` (per-country point counts, agreement %, country mismatches, worst
examples) and `mismatches.jsonl` (machine-readable list of all mismatches).
See `sweep --help` and `sample --help` for all options.

## License

This library is licensed under [the MIT license](https://github.com/lucaspiller/offline-geocoder/blob/master/LICENSE).
Expand Down
Loading
Loading