Search Search
Back to recaps

Week of June 1, 2026

The cross-entropy track lands in wikitext, the fixed-backward method gets a name, and a minimal optimizer benchmark called caffeine starts to form.

90 messages and 13 links in the archive this week.

A week that started quiet and grew three real arcs: the wikitext cross-entropy question got settled in code, the alternative-backprop thread picked up a name and an owner, and a debate about evaluating optimizers on random data started turning into a benchmark of its own.

The attention-optimizer thread

Yaroslav shared that he and a collaborator brainstormed an optimizer update meant to be efficient specifically for self-attention, rather than staying generic like Adam, with the working notes in a Google Doc. He framed the underlying question as how much room for improvement exists when using structure-specific solvers instead of gradient descent, and pointed to related outside work on curvature along gradient directions. Armins gave the practical constraint: the bottleneck is finding a toy problem where optimizing attention correlates with optimizing attention inside a real LLM training run. By Wednesday Yaroslav had stepped back from the full Transformer, calling it too complicated to read learning-rate interactions from, and moved to the smaller manifestai setup to iterate on solvers.

Cross-entropy becomes a track

The cross-entropy versus accuracy debate ran through midweek. Gabriel argued ngrams minimize cross-entropy too, and conditionally, so CE is a fair metric that captures what accuracy discards. Yaroslav countered that CE is a surrogate loss and asked why optimize it directly; Armins' practical case was that accuracy stops meaning much two or three words into a generation while perplexity stays informative, and that the way out of the argument was to try it. Gabriel did: the CE track PR landed Thursday, and under its threshold only nanoGPT passes. Every count-based method fails by a wide margin, and his agents found no workaround, so cross-entropy reads as a real performance bar. The earlier dev merge went to main the same week.

The fixed-backward method gets a name

Yaroslav unit-tested the fixed-backward-pass variant against regular backprop across learning rates after 1, 3, and 10 steps, in a comparison Colab, noting hyperparameters do not transfer between the two methods. Armins found the variant beats backprop at those horizons with relu, and got it converging once his sweep ran at 200 steps instead of 10 (his Colab). Two structural notes came out of the thread. The updates must run in order, layer 3 before layer 2 before layer 1, which the order-agnostic PyTorch optimizer interface cannot express; the proposed shape is a solver that takes (x, y) pairs and returns weights. And under a per-step automatic learning rate, classic backprop's rate stays flat while the variant's decreases over training. Yaroslav invited Gabriel to extend the method to cross-entropy loss with his Hessian math, offering the name nakaprop if it works at larger scale. Gabriel accepted, starting on the half-moon dataset, with MNIST plus feedforward layers suggested as the next step.

Caffeine, and the random-data question

The week's sharpest disagreement was whether random data can evaluate optimizers. Armins' position: random data is incompressible, so there is nothing for a learning algorithm to learn, the same way lossy image compression is tested on real images and not noise. His proposal was to run an autoresearch loop on the nanoGPT speedrun first, isolate what makes an optimizer good, then reduce that to a minimal task. Yaroslav's position: random data isolates architecture structure from data structure, and whatever improves fitting efficiency there should improve further once realistic data and data-specific tuning are added. His framing for the whole direction: given (x, y) pairs, recover a function from a family up to symmetry, and ask whether anything beats gradient descent for a specific family, with multigrid methods as an example of a non-gradient method specialized to a family. Josh Marks pointed at eigenvalue dispersion via the Distill momentum article. The thread closed with agreement to try different things and a new Telegram topic named caffeine for minimal optimizer tasks.

Learning-rate intuition, and two artifacts

A toy linear-attention experiment showed that if you tune each step's learning rate as high as possible without divergence, the stable rate grows over training. Yaroslav's explanation: late in training the residual is small, so gradients are small, so a much larger multiplier stays stable; his favorite related theory is that Muon works as an extra normalization that removes drift in effective step size. Alex Neagoe connected the stable rate to the inverse of the top Hessian eigenvalue. The experiments collected into a new repo, inverse-problems, with an interactive backprop visualizer: a 3-layer network learning a 90-degree rotation with every layer restricted to rotation matrices.

Side threads: Alex Varga traced his MacBook battery drain to Codex (post); Andy Zhang started a proof of concept on parallel plus asynchronous realtime agents, hit GPU limits running SSMs locally, and moved to Modal; Josh Marks asked for a weekend brainstorm and said he could make Monday's session.

Decisions recorded this week: the CE track is in, and the caffeine topic exists. The open questions carried forward: random versus real data for caffeine, and whether nakaprop converges at scale.

Sources