15 problems

The transformer architecture is used to estimate a context function $\mathbf{h}$, with smoothness index $\beta_2\leq 1$, and can represent a Nadaraya--Watson estimator. If the data…

Let $\mathcal{M}$ be a sufficiently capable base model trained on a corpus containing code, mathematics, and structured data, and let $n^*$ be a PLT trie node with probability…

$\mathsf{TC}^0$–$\mathsf{NC}^1$ separation conjecture. $\mathsf{TC}^0\neq\mathsf{NC}^1$. Under this widely believed conjecture, chain-of-thought computation would extend the expres…

Let $L$ be the length of a LEGO task, whose input consists of predicate and answer clauses, and let attention patterns describe which clauses a transformer uses when predicting the…

The BCJRFormer model processes received sequences of length $n_{\text{rec}}$ corresponding to codewords of length $n_{\text{in}}$, and uses attention across sequence tokens. Final-…

The attention heads in a transformer network carry information about relationships among input tokens. Attention-head information conjecture. Exploiting information carried by the…

Let a codeword have a given length, and let errors be introduced into it before correction. The error-ratio conjecture. The ratio of errors in the codeword is more closely related…

Let $\rho_\infty(\cdot,t)$ be the limiting parameter distribution at time $t$, and suppose that the origin $0_{\mathrm{dim}\theta+\mathrm{dim}w}$ is an interior point of…

A transformer is a neural architecture whose input can include an engineered prompt supplying additional information or memory. Prompt-engineering computational-power conjecture. P…

RASP length-generalization conjecture. Based on extensive empirical results, transformers tend to length-generalize on tasks that can be solved by RASP.

Let transformer-based neural networks be used as integrators for ordinary differential equations, in analogy with multi-step methods that can achieve arbitrary order of approximati…

Model-disorientation conjecture. Model disorientation leads randomly initialized models not to achieve their full potential, regardless of model size.

The GNOT attention block consists of a cross-attention layer followed by a self-attention layer; query points and input functions provide the information processed by these layers.…

Transformer rate-reduction conjecture. Layers of the Transformer emulate a more general family of gradient-based iterative schemes that optimize the rate reduction of all input tok…

The input consists of initial conditions, and the target consists of solutions at $t=1$; these are represented by the blue and green curves, respectively, in Figure. The model uses…