PhiloLM (opens in a new tab)
PhiloLM is an open language model (opens in a new tab) with 61 million parameters, trained from scratch on philosophy and psychology (opens in a new tab).
Advanced
How it works
PhiloLM is a decoder-only transformer with 16 blocks of width 576, the shape of SmolLM2-135M. Its design follows recent work on small models. A hashed bigram embedding from the modded-nanogpt speedrun (opens in a new tab) adds 31 million parameters that cost almost no computation. Three local attention layers alternate with one global layer, where 9 query heads share 3 key-value heads. Each head also has QK-norm, a sigmoid gate on its output and rotary position embeddings on half its dimensions.
It trained from scratch on PhiloText (opens in a new tab), 451 million tokens of philosophy and psychology. Training took 15 hours on one Intel Arc A770. Muon optimized the hidden matrices and AdamW the rest. The data was read in four passes that move from web pages to reference works, while the context grew from 1,024 to 2,048 tokens. The released weights blend the last weights with their exponential moving average from late training.
| Pass | Web pages | Books | Encyclopedias, reviews, textbooks and articles |
|---|---|---|---|
| 1 | 380.8 | 74.2 | 122 |
| 2 | 260.2 | 68.7 | 122 |
| 3 | 120.6 | 50.3 | 122 |
| 4 | 0 | 24.7 | 108.6 |
On the test split, PhiloLM predicts text at 0.944 bits per byte, where lower is better. On reference works and books, it beats Gemma 3 270M, which has three times its parameters and 4,000 times its training tokens. SmolLM2 and Qwen3, trained on 2 to 36 trillion tokens, do better, above all on web pages. PhiloLM's parameter count leaves out the bigram table, which is only looked up.
In your browser, an engine written in Rust and compiled to WebAssembly runs the model on your CPU. The page downloads 98 MB of 8-bit weights, which raise bits per byte by 0.2%. Up to four threads share the work. On a desktop processor, the page writes about 250 tokens a second. The model card (opens in a new tab) gives the details.