Skip to content

Add Laurens, IA — two imagery arms over one rural footprint, and the deficit is the rig, not the town (#149, #151) #152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jonfroehlich merged 11 commits into from
Sep 4, 2026
Merged

Add Laurens, IA — two imagery arms over one rural footprint, and the deficit is the rig, not the town (#149, #151) #152

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3e72084
Add Laurens, IA benchmark bundle — the first rural split
jonfroehlich Aug 31, 2026
5f3e12f
Pin Laurens imagery bytes before GT review
jonfroehlich Aug 31, 2026
517b3cf
Add Laurens GT verdicts — recall 0.390, the benchmark's worst by 2x
jonfroehlich Aug 31, 2026
0c8e0f4
Register Laurens as the 8th US split; pin two published studies
jonfroehlich Aug 31, 2026
8d61982
Rename laurens -> laurens_mapillary; it is now one of two arms
jonfroehlich Aug 31, 2026
62cc220
Add the Laurens GSV arm bundle -- the discriminator for #151
jonfroehlich Aug 31, 2026
e9dfd86
Laurens GSV arm GT: recall 0.505 -- #151 splits the difference
jonfroehlich Aug 31, 2026
5111834
Score all 11 challengers on laurens_mapillary; the AP test was a popu…
jonfroehlich Sep 1, 2026
d388f17
Register laurens_gsv as a held-out second arm: the Laurens deficit is…
jonfroehlich Sep 1, 2026
8312cbc
benchmark/README.md: cover both Laurens arms (#151, #152 Phase 6)
jonfroehlich Sep 4, 2026
c2c9121
claude-opus-5 on both Laurens arms: the strongest zero-shot model is …
jonfroehlich Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,257 changes: 1,257 additions & 0 deletions analysis_out/crop_window_eval_annapolis.json
View file
Open in desktop

Large diffs are not rendered by default.

1,261 changes: 1,261 additions & 0 deletions analysis_out/crop_window_eval_paterson.json
View file
Open in desktop

Large diffs are not rendered by default.

1,181 changes: 1,181 additions & 0 deletions analysis_out/crop_window_eval_sao_paulo.json
View file
Open in desktop

Large diffs are not rendered by default.

55 changes: 37 additions & 18 deletions analysis_out/op/confidence_calibration.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,25 @@ gainesville,0.8,0.85,21,2,0.9130434782608695,0.7320359570190598,0.97582050459276
gainesville,0.85,0.9,34,0,1.0,0.8984820937803899,1.0
gainesville,0.9,0.95,29,0,1.0,0.8830264055344442,1.0
gainesville,0.95,1.0,18,0,1.0,0.8241154494176252,1.0
laurens_mapillary,0.05,0.1,22,85,0.205607476635514,0.13988321742546095,0.2917381232985993
laurens_mapillary,0.1,0.15,11,33,0.25,0.14574066080685744,0.39440849808414946
laurens_mapillary,0.15,0.2,10,17,0.37037037037037035,0.21532287067936856,0.5577109537655369
laurens_mapillary,0.2,0.25,9,8,0.5294117647058824,0.30962897311549487,0.738351988039119
laurens_mapillary,0.25,0.3,12,5,0.7058823529411765,0.468664666737861,0.8672020613444361
laurens_mapillary,0.3,0.35,7,2,0.7777777777777778,0.4525833436728054,0.9367762376877727
laurens_mapillary,0.35,0.4,10,1,0.9090909090909091,0.6226353745137962,0.9837682477371741
laurens_mapillary,0.4,0.45,10,2,0.8333333333333334,0.5519636426153274,0.9530358523851776
laurens_mapillary,0.45,0.5,6,0,1.0,0.6096569663469354,0.9999999999999999
laurens_mapillary,0.5,0.55,1,1,0.5,0.09452865480086614,0.9054713451991339
laurens_mapillary,0.55,0.6,9,3,0.75,0.46768966087934005,0.9110599603710386
laurens_mapillary,0.6,0.65,11,2,0.8461538461538461,0.5776482949136899,0.9567427368172621
laurens_mapillary,0.65,0.7,13,3,0.8125,0.5699064013154177,0.9340852122641324
laurens_mapillary,0.7,0.75,11,2,0.8461538461538461,0.5776482949136899,0.9567427368172621
laurens_mapillary,0.75,0.8,9,0,1.0,0.7008472464490407,1.0
laurens_mapillary,0.8,0.85,20,0,1.0,0.8388698745050667,1.0
laurens_mapillary,0.85,0.9,14,1,0.9333333333333333,0.7018292244757601,0.9881334113938051
laurens_mapillary,0.9,0.95,10,0,1.0,0.7224598312333834,1.0
laurens_mapillary,0.95,1.0,0,0,,,
budapest_district5,0.05,0.1,7,107,0.06140350877192982,0.030058899274456374,0.12134434032853611
budapest_district5,0.1,0.15,7,52,0.11864406779661017,0.058675082779140006,0.22523875773415053
budapest_district5,0.15,0.2,11,39,0.22,0.12753786224302288,0.35241815689459555
Expand Down Expand Up @@ -189,22 +208,22 @@ manual_gold,0.8,0.85,326,13,0.9616519174041298,0.9355033815905722,0.977454672507
manual_gold,0.85,0.9,572,10,0.9828178694158075,0.9686628424978848,0.9906408327617758
manual_gold,0.9,0.95,984,12,0.9879518072289156,0.9790592412948215,0.9930947478970665
manual_gold,0.95,1.0,938,9,0.9904963041182682,0.9820370049703598,0.994992186852968
POOLED,0.05,0.1,45,485,0.08490566037735849,0.0640591684395796,0.11172630800137218
POOLED,0.1,0.15,31,181,0.14622641509433962,0.1049661102224358,0.2000798123430011
POOLED,0.15,0.2,29,110,0.20863309352517986,0.14937394144110333,0.2835643622463378
POOLED,0.2,0.25,30,77,0.2803738317757009,0.2040167905416766,0.3719546858715112
POOLED,0.25,0.3,26,52,0.3333333333333333,0.23872517005631883,0.4435879812017211
POOLED,0.3,0.35,27,32,0.4576271186440678,0.3370110214024243,0.583423849765719
POOLED,0.35,0.4,23,36,0.3898305084745763,0.2757917606558441,0.5173389043813288
POOLED,0.4,0.45,34,21,0.6181818181818182,0.4860897928239239,0.7348423368258477
POOLED,0.45,0.5,28,28,0.5,0.37331520958682995,0.62668479041317
POOLED,0.5,0.55,33,8,0.8048780487804879,0.6598611764744329,0.8976568425079408
POOLED,0.55,0.6,66,15,0.8148148148148148,0.7166820913952237,0.8844381137871395
POOLED,0.6,0.65,84,12,0.875,0.7941135793103388,0.9270287100760494
POOLED,0.65,0.7,94,8,0.9215686274509803,0.8528111615000344,0.959723796399317
POOLED,0.7,0.75,120,5,0.96,0.909773215091261,0.9827956446559014
POOLED,0.75,0.8,149,5,0.9675324675324676,0.9262531770462376,0.9860538446768062
POOLED,0.8,0.85,190,6,0.9693877551020408,0.934832884042845,0.9858963334974467
POOLED,0.85,0.9,335,4,0.9882005899705014,0.9700582252800168,0.9954022089263338
POOLED,0.9,0.95,297,1,0.9966442953020134,0.981239397663855,0.999407405162528
POOLED,0.05,0.1,67,570,0.10518053375196232,0.08367682864661602,0.13141782188793102
POOLED,0.1,0.15,42,214,0.1640625,0.12372695108681683,0.21433131210121778
POOLED,0.15,0.2,39,127,0.23493975903614459,0.17689210933867777,0.30497805674548534
POOLED,0.2,0.25,39,85,0.31451612903225806,0.2394074548604695,0.4007722675460084
POOLED,0.25,0.3,38,57,0.4,0.30722742667775993,0.5005458185954855
POOLED,0.3,0.35,34,34,0.5,0.3843786388809909,0.6156213611190091
POOLED,0.35,0.4,33,37,0.4714285714285714,0.35904683190771935,0.5867831609418397
POOLED,0.4,0.45,44,23,0.6567164179104478,0.5373244120046358,0.7591115804066028
POOLED,0.45,0.5,34,28,0.5483870967741935,0.42532294530598713,0.6658048523173997
POOLED,0.5,0.55,34,9,0.7906976744186046,0.6479406820576336,0.8857729400218853
POOLED,0.55,0.6,75,18,0.8064516129032258,0.7146734140699786,0.8739166133562513
POOLED,0.6,0.65,95,14,0.8715596330275229,0.7958940699649537,0.9219263083352476
POOLED,0.65,0.7,107,11,0.9067796610169492,0.8407592905282243,0.9471489444260114
POOLED,0.7,0.75,131,7,0.9492753623188406,0.8989997248371694,0.9752148920312246
POOLED,0.75,0.8,158,5,0.9693251533742331,0.9302094732086588,0.9868280042549953
POOLED,0.8,0.85,210,6,0.9722222222222222,0.9407323316014449,0.9872085130794528
POOLED,0.85,0.9,349,5,0.9858757062146892,0.9673668524828302,0.993952345732246
POOLED,0.9,0.95,307,1,0.9967532467532467,0.981840759704191,0.9994266529822499
POOLED,0.95,1.0,153,0,1.0,0.9755064982759676,1.0
Loading
Loading