31 problems

Gradient accumulation uses large-batch samples for gradient evaluation. Gradient-accumulation generalization conjecture. Although gradient accumulation can help during optimization…

For each positive integer $k$, a P-position of $k\times n$ Chomp consists of a weakly decreasing sequence of row lengths. Unique-extension generalization conjecture. If the Unique…

Consider the estimator $\widehat{\mathscr{E}}_{\texttt{test}}^{(t)}(X,Y)$ constructed by Algorithm; Theorem proves its consistency under the paper's single-index or multi-index reg…

Let $sigma_\alpha$ be the activation functions used in the neural network, and let $sigma'_\alpha$ and $sigma”_\alpha$ denote their first and second derivatives. Theorem establishe…

NASM is a neural adaptive spectral method whose architecture explicitly disentangles coefficients from basis functions. NASM generalization conjecture. We conjecture that NASM's ab…

RASP length-generalization conjecture. Based on extensive empirical results, transformers tend to length-generalize on tasks that can be solved by RASP.

A modular polynomial on $\mathcal{GF}(p)$ in two variables $(n_1,n_2)$ is said to satisfy the weak generalization hypothesis when it has the form … where $g_1$, $g_2$, and $h$ are…

Let a model be recalibrated using the same training data on which the model was trained, and let the resulting calibration be evaluated by an expected calibration error or related…

Large learning rates are considered in the context of gradient-based training, where a flat minimum is a minimum associated with favorable generalization properties. Flat-minima co…

Consider training neural networks with quadratic regularization terms. Loshchilov–Hutter conjecture. The quadratic regularization terms contribute to the low generalization error i…

Let $\beta$ and $\rho$ denote the momentum and second-moment decay parameters, respectively, in Adam. Consider a stable regime of training, and let the test error be the resulting…

The paper considers topological parallax as a comparison between a trained model and a reference dataset based on whether they have similar multiscale geometric structure. Topologi…

Consider large neural networks trained by an optimization process, their local minima, and their generalization performance on a test set. Equivalent-minima conjecture. Large neura…

Let stochastic gradient descent (SGD) use a learning rate and a batch size, and consider the width of the endpoint reached by the optimization process and its generalization capabi…

Let a deep neural network (a deepnet) be trained to a converged solution, and consider the curvature of its loss at that solution. Hochreiter et al.'s conjecture. The generalizatio…

A model is trained by a simple iterative method such as Stochastic Gradient Descent (SGD), and a flat local minimum is a local minimum associated with good generalization. Flat-min…

Let an empirical risk minimization procedure use a stochastic learning algorithm, and call a minimizer generalizing when it performs well beyond the training data. Implicit-bias co…

Overparameterized deep networks may have infinitely many global minimizers that fit the training samples exactly, so the optimization algorithm can determine which solution is obta…

Let $\mathcal{D}_{train}$ and $\mathcal{D}_{test}$ be the training and test subsets of the dataset, and let $D_{model}\circ E_{model}$ be a trained autoencoder. Near each training…

Memorization conjecture. The later deterioration is caused by the networks memorizing the answers for all the $\sigma$ values except $5$, thereby degrading performance at…

Let $N$ be the number of SGD iterations, $gamma$ the constant stepsize, $wb_0$ the initial iterate, $wb^*$ the population-risk minimizer, and $Hb$ the data covariance matrix. Write…

In the setting where training and test data points are drawn from the same distribution, the most salient features are predictive in both sets. Gradient Starvation's overfitting co…

A neural network estimator is obtained by optimizing an empirical loss over a class of deep neural networks. Optimizer-induced implicit regularization conjecture. It is conjectured…

Let a search path be a sequence of parameter values generated during optimization, and let its local Lipschitz constant measure the local smoothness along that path. Small-local-Li…

Mini-batch performance monotonicity conjecture. There exists a decreasing property for the expected loss, error and the generalization ability with respect to the mini-batch size.