Limits RegistryPUBLIC RECORD
Limits RegistryThe Switch Transformer, developed by William Fedus, Barret Zoph, and Noam Shazeer at Google, used a simplified mixture-of-experts routing scheme (each input token activated only one, rather than several, expert sub-networks) to reach 1.6 trillion total parameters while each individual token's forward pass used only a fraction of that capacity. It demonstrated large gains in pre-training speed over dense models of comparable computational cost.
The question, scope, and sources behind this Registry record.
The Switch Transformer, developed by William Fedus, Barret Zoph, and Noam Shazeer at Google, used a simplified mixture-of-experts routing scheme (each input token activated only one, rather than several, expert sub-networks) to reach 1.6 trillion total parameters while each individual token's forward pass used only a fraction of that capacity. It demonstrated large gains in pre-training speed over dense models of comparable computational cost.
Google's Switch Transformer, described in a January 2021 paper, scaled a sparsely activated mixture-of-experts transformer to 1.6 trillion parameters, the first publicly documented language model to cross the trillion-parameter mark, while keeping the computational cost per input comparable to much smaller dense models.
Change a parameter to stress-test whether a proposed result is still inside the published specification. This is an audit aid, not a proof checker.
This record has no editable parameters. Read the formal question and assumptions before challenging it.
Current frontiers derived from accepted Claims.
An observed or demonstrated result; no opposing bound is implied.
The frontier is not sacred
Most progress starts with a disagreement that survives contact with evidence. If you can push the known lower bound up or pull the upper bound down, show us the work.
≥ when you have shown that at least this value is achievable.≤ when you have shown that anything above this value is impossible.No vibes. State the value, define the scope, and link the paper, proof, code, or reproduction that lets another person check it. Editors review every challenge before the public record changes.
Challenge this recordAssertions tied to evidence, attribution, and review.
The frontier as it changed over time.
Only accepted Claims matching the current specification contribute to the displayed bounds. Strict inequalities remain open; contradictory Claims require editorial review.
1 accepted Claim, with 1 linked evidence records.
Permanent ID limitsregistry.com/limits/LR-SWITCH-TRANSFORMER-PARAMETERS
No active verified bounties are linked to this Limit.
View verified bounty tracker ↗No accepted machine-checked reproductions are recorded for this Limit.
Limits Registry. LR-SWITCH-TRANSFORMER-PARAMETERS. Switch Transformer — first publicly documented trillion-parameter model. 2026.