AI.

Exact self-attention naively materializes a score matrix that grows quadratically with sequence length, which is what limits how long a context transformer models can practically handle. IO-aware kernels like FlashAttention avoid materializing that full matrix, trading a fixed compute cost for dramatically less memory traffic.

Training a fixed compute budget into an ever-larger model isn't always the best use of that compute — the Chinchilla study found that model size and training-token count should scale together, not model size alone, to reach the best loss for a given budget.

Long-context language models don't retrieve information from anywhere in their context window with equal reliability — accuracy varies with where the relevant fact sits, how many distractors surround it, and which model and prompt are used, so no single model-independent floor can be asserted.

G key/value groups interpolate cache size between multi-query and full multi-head attention.

IO-aware tiling computes exact attention without materializing the quadratic score matrix.

HBM traffic is asymptotically optimal over the SRAM-size range analyzed in the paper.

Empirical compute-optimal parameter and token allocation for the model families and data regime studied in 2022.

Exact minimum hidden width for universal Lp approximation under the paper's ReLU architecture.

Fully connected ReLU networks no wider than the input dimension fail universal approximation.

Empirical neural language-model loss scaling with model size, dataset size, and compute in the studied regime.

A single shared key/value head makes per-layer KV cache linear in context length and head width, independent of query-head count.

Stored keys and values grow linearly in layers, context tokens, KV heads, and head width.

Standard permutation-invariant message-passing GNNs cannot distinguish graphs that one-dimensional Weisfeiler-Leman cannot distinguish.

With injective aggregation and sufficient capacity, GIN matches one-dimensional Weisfeiler-Leman distinguishing power.

Non-injective neighborhood aggregation loses distinctions available to the 1-WL test.

Upper capacity bound for piecewise-linear networks with W parameters and L layers.

Constructive lower capacity bound for piecewise-linear networks over the stated parameter range.

Tight parameter-unit capacity order for networks with W parameters and U nonlinear units.

Arbitrarily deep residual networks with one-neuron hidden layers are universal L1 approximators.

Exact dense dot-product attention forms all n-by-n query-key interactions of width d.

Explicit storage of the dense attention score or probability matrix grows quadratically with sequence length.

Summing per-step attention over a growing cached context gives quadratic total interaction work.

A k-layer local message-passing network depends only on each node's k-hop computation neighborhood.

Minimum iid labeled examples for distribution-free realizable PAC learning of VC dimension d.

Optimal distribution-free sample order for agnostic binary classification with VC dimension d.

Maximum affine regions cut by n ReLU hyperplanes in d-dimensional general position.

UCB1 has logarithmic instance-dependent expected regret for bounded iid arm rewards.

Posterior expected risk is controlled by empirical risk and divergence from a data-independent prior.

Averaged uniformly over all objective functions, optimization algorithms have identical performance.

Weighted majority competes with the best of N experts under bounded losses.

Tabular Q-learning converges under finite MDP, sufficient exploration, bounded rewards, and Robbins-Monro step sizes.

Finding weights consistent with labeled examples is NP-complete even for the specified three-node network.

Online halving makes at most logarithmically many mistakes when a finite class contains a perfect hypothesis.

Winnow learns a monotone disjunction with k relevant variables among n attributes with logarithmic dimension dependence.

Optimal deterministic realizable online mistake complexity equals the class Littlestone dimension.

Sufficient realizable-case sample size for a consistent learner over a finite hypothesis class.

Sample order sufficient for uniform convergence over a finite binary hypothesis class.

A VC-dimension-d class cannot realize more labelings than the binomial growth bound.

Perceptron mistakes are bounded for linearly separable examples of norm at most R and margin gamma.

The Bellman optimality operator is a gamma contraction for a discounted finite MDP.

Value iteration converges geometrically under the discounted Bellman contraction.