A plain transformer, 1.4 billion parameters.
The same design as our 103M yardstick, over thirteen times larger. It reads 2,048 tokens at a time and predicts the next one.
One block, repeated 30 times
Each token's vector goes through 30 identical blocks. In each block, attention lets the token look back at the tokens before it, and the MLP transforms it on its own. A residual line carries the vector past both, and each adds its result back in.
RMSNorm · rotary positions · QK-norm · grouped-query attention · SwiGLU
Most of the model is the MLPs
The 30 MLPs hold 72.5% of the weights. Attention holds 22.6%, and the shared embedding table, which also reads out the next token, holds 4.8%. Norm weights add 132,608 more.
Against our other models
Each square's area is its parameter count. Our all-SDM base and our transformer yardstick are about 100M each. XL is over thirteen times the yardstick.
The first model training
The first model trains on one DGX Spark, which has 121 GB of memory shared by its CPU and GPU. The model's fixed state takes about 19.7 GB. The rest is working memory for training, which the first timing run will measure.
| Weights, fp32 | 5.56 GB |
|---|---|
| Gradients | 5.56 GB |
| Muon momentum | 5.29 GB |
| AdamW state | 0.54 GB |
| bf16 weight copies | 2.78 GB |
| Fixed state | about 19.7 GB |