27 problems

Let $X$ be the data manifold formed by a collection of token embeddings in a $1536$-dimensional embedding space, and let $N$ denote the number of sampled token embeddings. Full-dim…

Let the ambient input space be the space containing real-world datasets, and let the intrinsic dimensionality of a dataset mean the dimensionality of the manifold on which it lies.…

Let the final-layer embeddings of a deep network be represented by their Gram matrix, and measure its complexity by effective rank. Huh et al. observe that this Gram matrix has low…

Let $\Omega^{\mathrm{o}}=\{(\mu,\nu)\in(0,1)\times(0,1)\mid \nu<\mu\}$ be the interior of $\Omega$, and let $J$ be the Jacobian defined in Eq.. Negativity conjecture. For every…

Smooth semantic trajectory conjecture. Factual content exhibits smooth, convergent trajectories in representation space, progressively aligning with ground-truth semantic embedding…

Stratified latent-space conjecture. Distinct strata in the latent space correspond to different state-action trajectories, with increases in local dimension occurring when the agen…

Redundant-encoding conjecture. More redundant encodings may be preferable for the network, as they could lead to representations that are more robust to small input variations and…

Information-bottleneck explanation. The underlying reason why training regularization has inferior downstream performance is that though it lowers the value of $I(Z_1;Z_2)$, it fai…

Let $A$ denote an image transformation, $C$ denote class information, and $X$ denote the transformed image. A transformation is global when it changes pixel positions, whereas it i…

HyperCube regularizer minimizer conjecture. The unitary group representation $A_g=B_g=C_g^\intercal=\varrho(g)$ describes the unique minimizer of $\mathcal{H}$ up to unitary basis…

Weight-feature equality conjecture. Then

Self-duality conjecture. The class feature aggregate is proportional to its corresponding classifier weight: for each $k=1,\ldots,K$, $\boldsymbol{h}_k^*=\alpha\boldsymbol{w}_k^*$…

Within-class collapsing. All features within each class are identical: for each $k=1,\ldots,K$, $\boldsymbol{h}_{k,i}^*\equiv\boldsymbol{h}_k^*$ for every $i$.

Intrinsic-dimension conjecture. The optimal value of $k$ is related to the intrinsic dimension of the data distribution, even when the distribution is not strictly supported on a l…

Let a neural network be trained on raw data to make predictions, and let topological features be features extracted from the underlying data using methods from topological data ana…

Structured-representation conjecture. Only on top of compact and structured representations learned by individual agents can the emergence and development of high-level intelligenc…

Transformer rate-reduction conjecture. Layers of the Transformer emulate a more general family of gradient-based iterative schemes that optimize the rate reduction of all input tok…

Parsimony conjecture for reinforcement learning. If reinforcement-learning objectives can be formalized more clearly as seeking certain low-dimensional correlations between state/a…

GROOS selection conjecture. Centered GROOS tends to perform better on mildly OOD datasets such as CIFAR100, Aircraft, and Fruits, whereas the simpler MinDist model might be a bette…

Consider a colored-digit environment in which ERM and spectral decoupling (SD) learn both digit and color features, while the color feature is inversely correlated with the label.…

Cartpole Balance Sparse conjecture. For the Cartpole Balance Sparse task, the $p(s'|s)$ model is still able to learn information as useful for achieving the goal as the…

Transfer-asymmetry conjecture. Solving the original Walk task would require exploring a wider range of the environment dynamics that presumably includes much what the Stand task wo…

Generalization conjecture. The information conveyed by robust representations has better generalization, and generalization is more of a problem on CIFAR-10 than on MNIST.

Let $\mathcal{Y}$ be the set of classes in a classification problem, and let $|\mathcal{Y}|$ denote the number of classes. A phase transition is a transition in the information bot…

Competing conjectures. In this regime, one conjecture is that spurious local minimizers exist, but descent methods with random initializations implicitly regularize themselves so t…