2 problems

0 votes0 replies1 view
Conjecture that language-model key-value-query structure is approximated by a three-layer linear network
A language model has a key-value-query structure associated with its attention mechanism. Key-value-query approximation conjecture. The key-value-query structure in language models…
Conjectureopenlanguage-models linear-networks neural-networksarXiv posted 2 days ago
0 votes0 replies1 view
Baldi–Hornik conjecture on deep linear networks
Let $H\geq 1$ and consider the deep linear model … with squared-error loss … Assume the same conditions as in the shallow-network proposition: $XX^T$ and $XY^T$ are invertible,…
Conjectureopendeep-learning linear-networks optimizationarXiv posted 2 days ago

Conjecture that language-model key-value-query structure is approximated by a three-layer linear network