The agreement property for independently trained classifiers

From papers

Let F\mathcal{F} be a classifier family and let D\mathcal{D} be a distribution. For independent train sets S1,S2DnS_1,S_2\sim\mathcal{D}^n, let f1,f2f_1,f_2 be classifiers trained on S1,S2S_1,S_2, respectively. The test accuracy is the probability that f1(x)=yf_1(x)=y, while the agreement property. For certain classifier families F\mathcal{F} and distributions D\mathcal{D}, these probabilities are approximately equal:

Pr[f1(x)=y]Pr[f1(x)=f2(x)].\Pr[f_1(x)=y]\approx\Pr[f_1(x)=f_2(x)].

Moreover, with high probability over the trained classifiers f1,f2f_1,f_2, the corresponding probabilities over (x,y)D(x,y)\sim\mathcal{D} are approximately equal. This is proposed as an instantiation of the paper's indistinguishability conjecture; it is provable for 1-nearest neighbors in some settings, but the general claim's resolution is not specified.

Progress summary

Nothing recorded yet. Refresh searches the literature and the public web for attempts on this problem, and writes the first summary here.

Sources & referencesView supporting material

Primary source

Preetum Nakkiran and Yamini Bansal, “Distributional Generalization: A New Kind of Generalization”, arXiv:2009.08092 (2020).

Solutions 0

No solutions have been posted yet.