Week of February 16, 2026
Energy-per-token thesis sharpens while makemore and pebble-game experiments push for a faster iteration loop.
235 messages and 32 links in the archive this week.
A heavy week, 235 messages, mostly in General and chat-yaroslav. The center of gravity was efficiency: how much energy training actually costs, and how to measure progress against it.
What moved
The makemore task drove most of the activity. Andy shared a one-shot prompt attempt in a Colab notebook, built on Karpathy's makemore gist and names.txt, and a follow-up run on CC plus GLM5. Seth posted his own iteration and then a larger run on roughly 32k of training data versus 1k. Daria asked whether accuracy could be plotted against energy spent.
Yaroslav flagged the iteration-speed problem directly: the MicroGPT experiments showed even the names task is too heavy to iterate on quickly, with one agent-loop iteration taking about three minutes, and noted the need for something faster. Anish pushed back that three minutes is not bad if you are still bottlenecked on agent tokens. As a faster proxy, Yaroslav pointed to the Red-Blue Pebble Game, a compilers problem for minimizing memory-access cost. Andy followed up by having GLM5 build a mini-game plus optimizer from that idea, with the task prompt and a PR.
Gabriel reported truncated-backprop experiments on a small model, propagating gradients through only the last 8 of 12 layers, claiming a 27% improvement in intelligence per joule, and later shared a Google Drive folder with scripts, workbook, logs, and a memo. Seth asked for a workbook to replicate.
The thesis
Much of chat-yaroslav and the Pitch / Talking Points topic circled the energy-cost argument. Yaroslav cited an arXiv paper observing training costs dropping 50% every 8 months, and his own imagenet18 work where a run went from $2000 to $40, a 50x reduction. He pointed at Jeff Dean on pico joules and made slides from a shared substack post. For the pitch, the working heuristic was that electricity is about 50% of total cost of ownership, sourced from Sequoia's AI's $600B Question, and that energy-to-train is gaining prominence again, with the note that it peaks once CFOs start balancing budgets. Gabriel framed the open question as the golden middle between two extremes, energy on one side and model dependency on the other.
GLM-5 came up in connection with RL time and expense, via an arXiv PDF and a shared Claude response; Seth quoted its asynchronous RL infrastructure decoupling generation from training. Yaroslav also noted that reports still need a human in the loop: due diligence on Asterlab's technical report using Deep Think 2.0 came back opinionated and wrong.
Decisions and open questions
Gabriel laid out a staged path for the pitch: train a 7B model on the new stack, validate, scale to 70B, validate, then go for frontier, arguing the cost of incremental validation is far less than staying on the current trajectory. Andy worked through how to frame the optimization target with an agent, landing on scheduler first rather than scheduler plus algorithm. Open threads included whether the makemore conclusion was even valid, what confidence interval to put on the RL share of training, and backing up the 5% useful math versus 95% overhead figure that anchors the pitch.
Sources
- arXiv:2403.05812, training costs dropping 50% every 8 months.
- arXiv:2602.15763, the GLM-5 RL cost reference.
- imagenet18, Yaroslav's $2000 to $40 training run.
- Red-Blue Pebble Game, the faster iteration proxy.
- gpu-clock-opt mini-game, Andy's GLM5 build from that idea.
- Karpathy's makemore gist, the base for the names task.
- Andy's Colab notebook, the one-shot prompt attempt.
- Seth's larger run, roughly 32k of training data versus 1k.
- Gabriel's Drive folder, truncated-backprop scripts, logs, and memo.
- Sequoia's AI's $600B Question, the electricity share of total cost, plus the links inline above.
- Telegram archive, week of February 16, 2026, paraphrased rather than quoted.