Worst Urban/Rural Subpop.

[niki-amini-naieni]

Hello,

I am wondering if you could clarify the following statement:
"We measure the average correlation, to test generalization under country shifts, and also the lower of the correlations on the urban and rural subpopulations, to test generalization between urban and rural subpopulations" (p. 103 of original paper)?

I am wondering which of the following calculations you report for Worst Urban/Rural Subpop.

1. min(average correlation for urban images over the five different folds, average correlation for rural images over five different folds)
2. Or do you choose the minimum correlation for urban images vs rural images in each fold and then average the five different pre-selected minimums?

Link to original paper: https://arxiv.org/pdf/2012.07421.pdf

Thank you!

[kohpangwei]

Good question! We do the latter (choose the minimum in each fold, then average the five different pre-selected minimums), though the two should be pretty similar since rural performance is typically worse. Sorry it was unclear.

[niki-amini-naieni]

Great, thank you!