Optimal chunking at surprisal peaks

Let $t_i$ be the token at position $i$, let $H_i=H(t_i\mid t_{<i})$ be its conditional entropy, and let $\mathcal{T}$ be the LAWS trie with description length $L(\mathcal{T})$. Optimal chunking at surprisal peaks. The optimal hierarchical decomposition of a LAWS expert library, minimizing total description length under a block-decomposition constraint, places chunk boundaries at positions $i$ where $H_i$ is locally maximal.

The sketch relates the gain from a boundary to $I(t_i;$expert label$\mid t_{<i})$ and identifies a greedy split with Huffman coding. Binary block decomposition is asserted to be covered by a known optimality result, while the extension to $k$-ary splits remains to be established.