C-RASP length-generalization conjecture and sample-size bounds
Determine asymptotically tight worst-case sample-size bounds for length generalization by transformers whose solutions are expressible in the fragments and . In particular, decide whether the previously known double-exponential upper bounds are tight, or whether substantially smaller bounds suffice.
References
Primary source
Additional references
Progress summary
A recent paper claims much tighter limits for how far these sequence models generalize, but the full conjecture and sample-size question remain open.
The problem concerns length generalization and sample-size bounds for and equivalent transformers. A March 2026 paper claims the unrestricted computability question has a negative answer, while a September 2026 paper claims substantially sharper quantitative bounds.
Known results
- One-layer : lengths suffice when parameter precision is bounded by (June 2025).
- Two-layer with at most heads: upper bound (June 2025).
- The unrestricted problem was claimed noncomputable, even at depth two; was claimed to have exponential upper and lower bounds, while sample-size questions remained open (March 2026).
- Learning a narrow teacher implementing a program was given sample bound under stated assumptions (July 2026).
September 2026 quantitative-bound claim
On September 8, 2026, Length Generalization for Transformers via Compression reported an exponentially smaller bound than the prior double-exponential estimate and claimed to reconcile conflicting experiments. It does not establish every form of the original hypothesis, and the claim has no independent verification in the retrieved sources.
Current status (as of September 2026): Restricted bounds are known, while the general computability claim and sharper quantitative bounds remain unverified and the full sample-size question remains open.
Sources
- arxiv.org
- arxiv.org
- arxiv.org
- arxiv.org
- hal.science
- icml.cc
- alphaxiv.org
- aaltodoc.aalto.fi
- proceedings.iclr.cc
- digital.lib.washington.edu
- openreview.net
- arxiv.org
- arxiv.org
- mathstodon.xyz
- mathstodon.xyz
- mathstodon.xyz
- mathstodon.xyz
- mathstodon.xyz
- community.openai.com
- quantamagazine.org
- scientificamerican.com
- cdn.openai.com
Solutions 0
No solutions have been posted yet.