04 / Transformer language modeling · NanoGPT
Inside the language model.
Three small GPTs, trained the same way, measured the same way.
Read the diagram
- Data preparation
- Leakage-free BPE
- Three matched GPT runs
- Token-weighted perplexity
01 / Context
The problem
Comparing language models is easy to get wrong. If the tokenizer has already seen the test text, or the evaluation averages the wrong way, the scores stop meaning much, however good the training run was.
02 / Contribution
What I did
I trained three GPT models of 29.9 million parameters each under matched hyperparameters, and built the tokenisation and evaluation pipeline around them.
03 / Decisions
The approach
I built a byte-pair encoding pipeline that never sees the evaluation data, then wrote a custom perplexity metric weighted by token count. Matching the hyperparameters keeps the runs comparable. Weighting by tokens means every token counts the same, however the batches happen to split.
- Data preparation
- Leakage-free BPE
- Three matched GPT runs
- Token-weighted perplexity
How the main pieces fit together.
04 / Evidence
What I can show
What exists: three trained models and the pipeline that prepares their data and scores them. I’m not publishing perplexity figures or a ranking on this page.
05 / Reflection
Limits & lessons
Model size and run count describe the setup, not the quality, and the code, data and training curves for this one aren’t public. The lesson that stuck: make the comparison trustworthy first, then look at the score.