May 2026
Yad's wave-build swarms ship the hinton and schmidhuber catalogs, the lab spreads into four challenges, and wikitext opens a language-modeling front.
In May the research moved out of SutroYaro into sibling repos, and the work reorganized around four parallel challenges plus two catalogs that agents built almost end to end. The month's biggest piece of engineering was Yad's: two wave-build swarms that reproduced decades of papers as runnable stubs. This recap draws on the changelog (v0.30.0 to v0.31.0), the May 20 catch-up, and the Telegram archive (now covering all 11 forum topics).
Yad's wave-build catalogs
Yad ran two agent swarms with the wave-build method, and each produced a catalog of runnable paper reproductions:
- hinton-problems: 53 pure-numpy stubs of Hinton's representational work, shipped May 3 across 11 waves. 27 reproduce the paper result, 25 partial, 1 honest non-replication.
- schmidhuber-problems: 58 stubs of Schmidhuber's algorithmic work, shipped May 8 across 12 waves. 32 reproduce, 25 partial, 1 non-replication. About 41 hours of wall time and roughly 1.15 billion tokens.
The method: Yaroslav specifies a stub set, Yad's agents reproduce it in parallel waves (one agent per stub, fresh teammates each wave), Yad gates the merges with an audit agent per wave, and a classified catalog falls out. One recurring point from Yaroslav: because agents emit unlimited text while human input stays sparse, the load-bearing human prompts should be captured in every PR. On the Schmidhuber run, 8 of 40 driver prompts were load-bearing, at about 25.7 lead turns per prompt.
The four challenges
The research front spread into sibling repos.
| Challenge | What | Repo | State in May |
|---|---|---|---|
| #1 Sparse parity | The original benchmark (ByteDMD metric) | sparse-parity-challenge | Dormant |
| #2 Matmul | Energy-efficient matmul as an IR on a 2D grid | sutro-problems/matmul | Active leaderboard |
| #3 Sparse parity on the grid | The same task in about 9 grid instructions | sutro-problems/sparse-parity | Launched May 8 |
| wikitext (#4) | Energy-efficient language modeling on WikiText-103 | cybertronai/wikitext | Launched, very active |
Matmul became a real leaderboard contest: the 16x16 record fell from 68,452 (Sung Jae Bae, May 5) to 68,392 (Cosmin Negruseri, May 13) to 67,821 (May 14), with Cosmin running a Codex-driven auto-research loop that produced a steady stream of small PRs. Lower bounds proved hard. An agent-generated bound was wrong on inspection, and Yaroslav noted that even AlphaTensor could not lower-bound 4x4. Grid sparse parity (#3) is hard to cheat by design: the op set is so small (about 9 instructions) there is nowhere to hide free work.
wikitext: the language-modeling front
The newest challenge, led by new member Armins, is energy-efficient language modeling
on WikiText-103. Infrastructure moved to Modal with GPU energy measured through NVML;
the baseline modded_nanogpt runs in 322.7 s for 54,784 J at 0.7285
character-accuracy. The objective is to hold accuracy and wall-clock fixed and shrink
the Joules. An early sweep found Forward-Forward reaching 0.39 accuracy at roughly 10x
fewer Joules than the baseline, while state-space models had not converged. Gabriel
Nakajima An noticed that NVML only counts GPU energy, so CPU-heavy submissions can hide
work; the open question is whether to add CPU package energy (RAPL). Yaroslav's
position is total system energy, no device constraints, under time and accuracy bounds.
The reshuffle and the metric roadmap
SutroYaro was stripped to its lab-memory role (issue #96): research code migrated out to sparse-parity-challenge (PR #40) and the duplicates were removed (PR #97, merged May 14). Yad also fixed the Telegram sync, which had silently carried a hardcoded six-topic list, to cover all 11 forum topics, backfilling 385 messages. That is why this archive is now complete. A two-pass metric roadmap took shape over the catalogs: v2 is which stubs ByteDMD can instrument, v3 is which of those the Bill Dally 2D-grid model can instrument. The survivors become the next competition; Yaroslav floated skipping straight to v3.
People and meetings
New and returning faces: Armins (leading wikitext), Gabriel Nakajima An (energy metering), Cosmin Negruseri (matmul auto-research, now attending in person), Sung Jae Bae (matmul and grid parity), Louka Ewington-Pitsos (Forward-Forward), and Anastasiia Zhiboedova, who shared SutroAna, an agent harness for a focused "improve this one problem" loop used on the matmul challenge. The Monday cadence held: May 4 had a guest talk by Russ Pantone (ex-Rain.AI) on building AI chips, and the May 18 meeting in San Francisco walked through records progress. Yaroslav gave an AI Council talk on May 12.
Reusable this period
- Yad's wave-build method: multi-agent paper reproduction that emits a catalog of runnable, classified baselines.
- The hinton and schmidhuber catalogs themselves, both pure numpy.
- The task-scale ladder for choosing how hard a task to iterate on: matmul (fast), then Shakespeare and TinyStories, then WikiText-103, then FineWeb.
What's next
- The auto-research-loop dispatcher kit (v0.31.0, May 29) packages Yad's wave method so a second operator can drive it. See June.
- Instrument the catalogs with ByteDMD (the v2 pass).
- Resolve the wikitext CPU-energy and cross-entropy questions.
Sources
- Changelog: SutroYaro v0.30.0 to v0.31.0 (May 20 to 29).
- Catch-up: the Apr 30 to May 20 status update.
- Repos: hinton-problems, schmidhuber-problems, wikitext, sutro-problems, SutroAna; reshuffle #96 and #97.
- Telegram: 654 messages across all 11 topics, May 1 to 31.