Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
### Unreleased

- Keep counties out of the dominant-city rollup. A parent geohash cell rolled
up to the place that dominates its children on population, and `county`
counted as a city-like placetype, so a metro could be labelled with its
county's name while the smaller localities under it lost their own cells
(Brockport and Greece, NY answering "Monroe"). The rollup now only hands a
parent cell to a placetype listed in `--dominant-city-placetypes`
(default `locality,localadmin`, env `WOF_DOMINANT_CITY_PLACETYPES`); add
`county` to that list to reproduce earlier builds. Counties still win cells
through the normal comparator. Both routes into a parent cell -- winning the
dominant-city competition, and being the only city-like owner in the cell --
share one eligibility gate, so a county cannot take a parent cell through the
single-candidate path either.

- Document the database backwards-compatibility contract in COMPATIBILITY.md
and enforce it with permanent per-generation reader fixtures in
spec/reader_compatibility_spec.js
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,8 +223,8 @@ The builder uses a multi-stage pipeline to decide which localities make it into
2. **Isolation pass** (`--isolation-min-population`): localities between the isolation floor and the primary threshold are evaluated as candidates. A candidate is promoted if at least one of its geohash cover cells (at base precision) is not already claimed by a primary locality. This ensures small but geographically isolated places like islands, remote towns, and oases get their own label without adding noise in dense urban areas.
3. **Country guarantee** (`--ensure-country-locality`): after the isolation pass, any country that still has zero localities gets its highest-population candidate promoted unconditionally.
4. **Contained-locality pruning** (`--drop-contained-localities`): removes localities whose polygon is fully contained inside a larger locality in the same country/admin1 group.
5. **Dominant-city rollup**: in the geohash index, when a major city (population >= `--dominant-locality-population`) dominates its neighbours by a ratio of `--dominant-locality-ratio`, smaller nearby localities are absorbed into the major city label.
6. **Locality-over-region promotion**: when a locality and a region compete for the same parent geohash cell, the locality wins if it covers >= `--parent-locality-min-share` of child cells.
5. **Dominant-city rollup**: in the geohash index, when a major city (population >= `--dominant-locality-population`) dominates its neighbours by a ratio of `--dominant-locality-ratio`, smaller nearby localities are absorbed into the major city label. Only the placetypes listed in `--dominant-city-placetypes` (default `locality,localadmin`) may play that dominant role, so a metro is never named after the county it sits in; counties still own cells on their own merits.
6. **Locality-over-region promotion**: when a locality and a region compete for the same parent geohash cell, the locality wins if it covers >= `--parent-locality-min-share` of child cells and its placetype is listed in `--dominant-city-placetypes`. Both routes to a parent cell share that eligibility gate, so a lone county cannot take a parent cell that the dominant-city rollup would have refused it.

Builder notes:

Expand All @@ -242,6 +242,7 @@ Builder notes:
- Dominant-city rollup keeps broad city labels sticky in mixed city/suburb cells unless there is competing major-city pressure:
- `--dominant-locality-population` (default `100000`)
- `--dominant-locality-ratio` (default `3`)
- `--dominant-city-placetypes` (default `locality,localadmin`) placetypes eligible to be the dominant city of a parent cell, whether it wins a competition or is the only city-like owner in that cell. A county wins cells like any other place, but naming a whole parent cell after it reads as a mistake, so it is excluded by default; add `county` to reproduce pre-1.1 builds
- Parent-cell takeover guard:
- `--parent-locality-min-share` (default `0.5`) requires locality ownership of at least that child-cell share before replacing a parent cell label
- Excludes neighbourhood-like placetypes from default reverse output
Expand Down Expand Up @@ -282,6 +283,7 @@ Useful WOF build env vars:
- `WOF_PROMOTE_LOCALITY_OVER_REGION=1|0` prefer locality labels over region in shared parent cells (default `1`)
- `WOF_DOMINANT_LOCALITY_POPULATION` major-locality threshold for dominant-city rollup (default `100000`)
- `WOF_DOMINANT_LOCALITY_RATIO` dominant-vs-next locality population ratio (default `3`)
- `WOF_DOMINANT_CITY_PLACETYPES` placetypes eligible to be a parent cell's dominant city (default `locality,localadmin`)
- `WOF_PARENT_LOCALITY_MIN_SHARE` minimum child-cell share for locality parent takeover (default `0.5`)
- `WOF_GEOMETRY_DECIMALS` round coordinates before storage/indexing (for example `4`)
- `WOF_MIN_POPULATION` filter out places below threshold (for example `10000`)
Expand Down
89 changes: 85 additions & 4 deletions scripts/generate_boundary_index.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ const PLACETYPE_BY_CODE = {
3: 'county'
}

// Placetypes allowed to play the "dominant city" role in the parent-cell
// rollup. A county is a city-like placetype for ownership purposes, but naming
// a metro after its county reads as a mistake, so counties are out by default.
const DEFAULT_DOMINANT_CITY_PLACETYPES = ['locality', 'localadmin']

function parseBool(value, defaultValue) {
if (value === undefined || value === null || value === '') {
return defaultValue
Expand All @@ -38,6 +43,34 @@ function parseBool(value, defaultValue) {
return defaultValue
}

function parsePlacetypeList(value, defaultValue) {
if (value === undefined || value === null || String(value).trim() === '') {
return defaultValue
}

var parts = String(value).split(',')
var placetypes = []
for (var i = 0; i < parts.length; i++) {
var name = parts[i].toLowerCase().trim()
if (!name) continue
if (!Object.prototype.hasOwnProperty.call(PLACETYPE_CODES, name)) {
throw new Error('Unknown placetype in --dominant-city-placetypes: ' + parts[i].trim())
}
if (!isCityPlacetypeCode(PLACETYPE_CODES[name])) {
throw new Error('--dominant-city-placetypes only accepts city-like placetypes: ' + parts[i].trim())
}
if (placetypes.indexOf(name) === -1) {
placetypes.push(name)
}
}

if (!placetypes.length) {
throw new Error('--dominant-city-placetypes requires at least one placetype')
}

return placetypes
}

function parseArgs(argv) {
var opts = {
database: null,
Expand Down Expand Up @@ -68,7 +101,8 @@ function parseArgs(argv) {
promoteLocalityOverRegion: true,
dominantLocalityPopulation: 100000,
dominantLocalityRatio: 3,
parentLocalityMinShare: 0.5
parentLocalityMinShare: 0.5,
dominantCityPlacetypes: DEFAULT_DOMINANT_CITY_PLACETYPES.slice()
}

for (var i = 0; i < argv.length; i++) {
Expand Down Expand Up @@ -142,6 +176,8 @@ function parseArgs(argv) {
} else if (arg === '--dominant-locality-ratio') {
var dominantRatio = Number(argv[++i])
opts.dominantLocalityRatio = Number.isFinite(dominantRatio) ? dominantRatio : opts.dominantLocalityRatio
} else if (arg === '--dominant-city-placetypes') {
opts.dominantCityPlacetypes = parsePlacetypeList(argv[++i], opts.dominantCityPlacetypes)
} else if (arg === '--parent-locality-min-share') {
var minShare = Number(argv[++i])
opts.parentLocalityMinShare = Number.isFinite(minShare) ? minShare : opts.parentLocalityMinShare
Expand Down Expand Up @@ -194,6 +230,7 @@ function usage() {
' --dominant-locality-population Population threshold that marks locality as major for dominant-city rollups (default: 100000)',
' --dominant-locality-ratio Required dominant-vs-next population ratio for locality rollups (default: 3)',
' --parent-locality-min-share Minimum child-cell share (0..1) required to let a locality take over a parent cell (default: 0.5)',
' --dominant-city-placetypes Comma-separated placetypes eligible to take over a parent cell, by competition or as its only city-like owner (default: ' + DEFAULT_DOMINANT_CITY_PLACETYPES.join(',') + ')',
' --append Keep existing boundary rows and append/replace by place id',
' --replace Clear boundary rows first (default)',
' --help, -h Show this help message'
Expand Down Expand Up @@ -946,6 +983,28 @@ function isCityPlacetypeCode(code) {
return code === PLACETYPE_CODES.locality || code === PLACETYPE_CODES.localadmin || code === PLACETYPE_CODES.county
}

// Owning a cell and standing in for a whole metro are different jobs. Every
// city-like placetype (county included) can win a cell through the comparator;
// only these placetypes may become the dominant city a parent cell is named
// after, so a metro never ends up labelled with its county's name.
function isDominantCityPlacetypeCode(code, opts) {
if (!isCityPlacetypeCode(code)) {
return false
}

var names = opts && Array.isArray(opts.dominantCityPlacetypes) && opts.dominantCityPlacetypes.length
? opts.dominantCityPlacetypes
: DEFAULT_DOMINANT_CITY_PLACETYPES

for (var i = 0; i < names.length; i++) {
if (PLACETYPE_CODES[names[i]] === code) {
return true
}
}

return false
}

// Rebuild a comparable place record from a stored compact_places row so that
// appended batches can be ranked against places written by earlier batches.
// Databases created before population/area were stored yield NULL for those
Expand Down Expand Up @@ -1011,6 +1070,7 @@ function selectDominantLocalityId(localityIds, placeById, opts) {
var place = placeById[String(id)]
return {
id: Number(id),
placetypeCode: place ? place.placetypeCode : null,
population: placePopulation(place)
}
})
Expand Down Expand Up @@ -1065,6 +1125,24 @@ function localityShareMeetsThreshold(localityId, group, opts) {
return localityShareInParent(localityId, group) >= threshold
}

// The single gate for taking over a parent cell, shared by both roll-up paths
// -- the lone city-like owner and the winner of a dominant-city competition --
// so that neither can drift away from the other. A candidate qualifies only if
// its placetype may name a parent cell (counties may not, by default) and it
// owns enough of the parent's child cells to stand in for the whole of it.
// A candidate rejected here is not replaced by a runner-up: the runner-up lost
// the cell competition, so promoting it would name the parent after a place
// that owns less of it. The parent keeps its existing owner instead, and every
// child keeps the cell it won.
function localityMayTakeOverParent(localityId, group, placeById, opts) {
var place = placeById[String(localityId)]
if (!place || !isDominantCityPlacetypeCode(place.placetypeCode, opts)) {
return false
}

return localityShareMeetsThreshold(localityId, group, opts)
}

function promoteLocalityParentsByRegionCompetition(bestByHash, placeById, opts) {
if (!opts.promoteLocalityOverRegion) {
return
Expand Down Expand Up @@ -1123,7 +1201,7 @@ function promoteLocalityParentsByRegionCompetition(bestByHash, placeById, opts)

if (localityIds.length === 1) {
var localityId = Number(localityIds[0])
if (!localityShareMeetsThreshold(localityId, group, opts)) {
if (!localityMayTakeOverParent(localityId, group, placeById, opts)) {
continue
}

Expand All @@ -1148,7 +1226,7 @@ function promoteLocalityParentsByRegionCompetition(bestByHash, placeById, opts)
if (dominantLocalityId === null) {
continue
}
if (!localityShareMeetsThreshold(dominantLocalityId, group, opts)) {
if (!localityMayTakeOverParent(dominantLocalityId, group, placeById, opts)) {
continue
}

Expand Down Expand Up @@ -1865,6 +1943,9 @@ async function main() {
if (!Number.isFinite(options.dominantLocalityRatio) || options.dominantLocalityRatio < 1) {
options.dominantLocalityRatio = 1
}
if (!Array.isArray(options.dominantCityPlacetypes) || !options.dominantCityPlacetypes.length) {
options.dominantCityPlacetypes = DEFAULT_DOMINANT_CITY_PLACETYPES.slice()
}
if (!Number.isFinite(options.parentLocalityMinShare)) {
options.parentLocalityMinShare = 0.5
}
Expand Down Expand Up @@ -1963,7 +2044,7 @@ async function main() {
console.log('Dense county rule: area_km2<=' + options.countyDenseMaxAreaKm2 + ' => max_precision=' + options.countyDenseMaxPrecision)
}
if (options.dominantLocalityPopulation > 0) {
console.log('Dominant locality rollup: major_population>=' + options.dominantLocalityPopulation + ', ratio>=' + options.dominantLocalityRatio)
console.log('Dominant locality rollup: major_population>=' + options.dominantLocalityPopulation + ', ratio>=' + options.dominantLocalityRatio + ', placetypes=' + options.dominantCityPlacetypes.join(','))
} else {
console.log('Dominant locality rollup: disabled')
}
Expand Down
3 changes: 3 additions & 0 deletions scripts/generate_wof_boundary.sh
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ set -euo pipefail
# WOF_DOMINANT_LOCALITY_POPULATION Major-locality threshold for dominant-city rollup (default: 100000)
# WOF_DOMINANT_LOCALITY_RATIO Dominant-vs-next locality population ratio (default: 3)
# WOF_PARENT_LOCALITY_MIN_SHARE Minimum child-cell share (0..1) required for locality parent takeover (default: 0.5)
# WOF_DOMINANT_CITY_PLACETYPES Placetypes eligible to take over a parent cell (default: locality,localadmin)
# WOF_INCLUDE_LOCALADMIN Include localadmin placetypes (default: 0)
# WOF_INCLUDE_COUNTY Include county placetypes (default: 1)
# WOF_INCLUDE_REGION Include region placetypes (default: 1)
Expand Down Expand Up @@ -72,6 +73,7 @@ WOF_PROMOTE_LOCALITY_OVER_REGION="${WOF_PROMOTE_LOCALITY_OVER_REGION:-1}"
WOF_DOMINANT_LOCALITY_POPULATION="${WOF_DOMINANT_LOCALITY_POPULATION:-100000}"
WOF_DOMINANT_LOCALITY_RATIO="${WOF_DOMINANT_LOCALITY_RATIO:-3}"
WOF_PARENT_LOCALITY_MIN_SHARE="${WOF_PARENT_LOCALITY_MIN_SHARE:-0.5}"
WOF_DOMINANT_CITY_PLACETYPES="${WOF_DOMINANT_CITY_PLACETYPES:-locality,localadmin}"
WOF_INCLUDE_LOCALADMIN="${WOF_INCLUDE_LOCALADMIN:-0}"
WOF_INCLUDE_COUNTY="${WOF_INCLUDE_COUNTY:-1}"
WOF_INCLUDE_REGION="${WOF_INCLUDE_REGION:-1}"
Expand Down Expand Up @@ -151,6 +153,7 @@ COMMON_FLAGS=(
--dominant-locality-population "${WOF_DOMINANT_LOCALITY_POPULATION}"
--dominant-locality-ratio "${WOF_DOMINANT_LOCALITY_RATIO}"
--parent-locality-min-share "${WOF_PARENT_LOCALITY_MIN_SHARE}"
--dominant-city-placetypes "${WOF_DOMINANT_CITY_PLACETYPES}"
--include-localadmin "${WOF_INCLUDE_LOCALADMIN}"
--include-county "${WOF_INCLUDE_COUNTY}"
--include-region "${WOF_INCLUDE_REGION}"
Expand Down
Loading
Loading