PhiloLM⁠ (opens in a new tab)

PhiloLM is an open language model (opens in a new tab) with 61 million parameters, trained from scratch on philosophy and psychology (opens in a new tab).

How it works

PhiloLM is a decoder-only transformer with 16 blocks of width 576, the shape of SmolLM2-135M. Its design follows recent work on small models. A hashed bigram embedding from the modded-nanogpt speedrun (opens in a new tab) adds 31 million parameters that cost almost no computation. Three local attention layers alternate with one global layer, where 9 query heads share 3 key-value heads. Each head also has QK-norm, a sigmoid gate on its output and rotary position embeddings on half its dimensions.

Blocks
61.4M
Bigram table
31.5M
Token embedding
18.9M
The 16 blocks hold 61.4 million parameters. The bigram table holds 31.5 million more, read one row at a time, so it adds almost no computation.

It trained from scratch on PhiloText (opens in a new tab), 451 million tokens of philosophy and psychology. Training took 15 hours on one Intel Arc A770. Muon optimized the hidden matrices and AdamW the rest. The data was read in four passes that move from web pages to reference works, while the context grew from 1,024 to 2,048 tokens. The released weights blend the last weights with their exponential moving average from late training.

Training tokens per pass, in millions, by kind of text
PassWeb pagesBooksEncyclopedias, reviews, textbooks and articles
1380.874.2122
2260.268.7122
3120.650.3122
4024.7108.6
Training tokens in each pass over the data. Web pages give way to reference works, and the context doubles after the first quarter of training.

On the test split, PhiloLM predicts text at 0.944 bits per byte, where lower is better. On reference works and books, it beats Gemma 3 270M, which has three times its parameters and 4,000 times its training tokens. SmolLM2 and Qwen3, trained on 2 to 36 trillion tokens, do better, above all on web pages. PhiloLM's parameter count leaves out the bigram table, which is only looked up.

ModelWeb pagesReference worksBooks
PhiloLM80M, 1.45B tokens0.950.881.03
SmolLM2-135M (opens in a new tab)135M, 2T tokens0.800.840.96
LFM2.5-230M (opens in a new tab)230M, 28T tokens0.951.031.32
Gemma 3 270M (opens in a new tab)268M, 6T tokens0.870.941.13
SmolLM2-360M (opens in a new tab)362M, 4T tokens0.730.770.89
Qwen3-0.6B (opens in a new tab)596M, 36T tokens0.740.790.93
Bits per byte on PhiloText's test split, lower is better. Each dot sits on a scale from 0.7 to 1.35.

In your browser, an engine written in Rust and compiled to WebAssembly runs the model on your CPU. The page downloads 98 MB of 8-bit weights, which raise bits per byte by 0.2%. Up to four threads share the work. On a desktop processor, the page writes about 250 tokens a second. The model card (opens in a new tab) gives the details.