FLAGSHIP / MACHINE INTELLIGENCE
I kept pulling on the question of how language models actually work until there was a tokenizer, a training loop, checkpoints, evaluation, and a model family with version numbers.
The point is not to build a tiny ChatGPT. The point is to make the abstraction disappear.
CURRENT GENERATION
G4-v6 finished its full 1.63B-token pretraining run. The best checkpoint landed at step 793,750 with 2.5328 validation loss. Now that same 69.1M model is being instruction-tuned on 27,024 assistant examples so it can learn how to answer people, not just continue text.
WHY
Using a model is easy. Building one forces every hidden decision into the open: what data looks like, how text becomes tokens, what the network is allowed to remember, how optimization behaves, and how quickly a model can learn the wrong lesson.
Every answer creates three more questions.THE CURRENT WORK
G4-v6 completed all 796,545 planned optimizer steps across the 1.63B-token training corpus. The best checkpoint came from step 793,750 at 2.5328 validation loss. That checkpoint is now the base for response-only instruction tuning across 27,024 training examples and 524 validation examples.
The model learned language first. Now it learns the assistant role.WHAT I WATCH
The corpus is not just a giant text pile. It pulls from broad technical, educational, reference, web, and book sources, then filters duplicates, repetition, boilerplate, and low-quality patterns before training. More data only helps if the model is learning from something worth keeping.
WHERE IT GOES
The completed base model already has frozen behavior benchmarks. After instruction tuning, it goes through those again plus assistant-focused tests. That comparison should show how much of the remaining weakness is model capacity and how much was simply never teaching the base model how to answer a user.
G5 waits until G4-SFT earns the next move.PROJECT DNA
THE REST OF THE LAB