The inverse large sieve conjecture

Let $X\subset\mathbb{Z}$ be a subset and let $\epsilon>0$ be real. Assume that, for every prime $p\geq\epsilon^{-1}$, the reduction $X\pmod p$ occupies at most $0.99p$ residue classes. Inverse large sieve conjecture. Either $|X\cap[1,N]|\ll_{\epsilon}N^{\epsilon}$ for all sufficiently large $N$, or there exists a polynomial $f(x)\in\mathbb{Q}[x]$ of degree $d\geq2$ such that $X\setminus f(\mathbb{Q})$ is finite. The conjecture is intended as an infinitary inverse statement for the larger sieve: persistent omission of a positive proportion of residue classes should force either sparse growth or containment, up to finitely many exceptions, in a polynomial value set. The source immediately notes that this formulation is false, because examples can be formed by taking unions of value sets of several distinct polynomials.