The agreement property for independently trained classifiers

At least 5 years old · documented by

Let F\mathcal{F} be a classifier family and let D\mathcal{D} be a distribution. For independent train sets S1,S2∼DnS_1,S_2\sim\mathcal{D}^n, let f1,f2f_1,f_2 be classifiers trained on S1,S2S_1,S_2, respectively. The test accuracy is the probability that f1(x)=yf_1(x)=y, while the agreement property. For certain classifier families F\mathcal{F} and distributions D\mathcal{D}, these probabilities are approximately equal:

Pr⁡[f1(x)=y]≈Pr⁡[f1(x)=f2(x)].\Pr[f_1(x)=y]\approx\Pr[f_1(x)=f_2(x)].

Moreover, with high probability over the trained classifiers f1,f2f_1,f_2, the corresponding probabilities over (x,y)∼D(x,y)\sim\mathcal{D} are approximately equal. This is proposed as an instantiation of the paper's indistinguishability conjecture; it is provable for 1-nearest neighbors in some settings, but the general claim's resolution is not specified.

References

Primary source

Preetum Nakkiran and Yamini Bansal, “Distributional Generalization: A New Kind of Generalization”, arXiv:2009.08092 (2020).

Progress summary

Never refreshed

Nothing recorded yet. Refresh searches the literature and the public web for attempts on this problem, and writes the first summary here.

Solutions 0

No solutions have been posted yet.