Search Search
Back to recaps

Week of May 4, 2026

Schmidhuber-problems shipped and the wikitext challenge split into two tracks. Sparse parity got a 50 percent target.

199 messages and 37 links in the archive this week.

The big result was the schmidhuber-problems catalog going live, a companion to the earlier hinton-problems run. Yad fired off the agent swarm Yaroslav had stubbed out and reported the finished site mid-week: roughly 41 hours of wall time and about 780k tokens, slower than the hinton run because Yad was only checking in every 18 hours or so while doing other things. The summary was 12 wave PRs all merged, 32 reproduce-paper-claims, 25 partial or qualitative, and 1 honest non-replication (hq-learning-pomdp, with the gap analyzed in the open questions section). Site at cybertronai.github.io/schmidhuber-problems, repo at github.com/cybertronai/schmidhuber-problems. Yaroslav asked Yad to swap the homepage "Site" link for a "Github" link pointing at the repo. The earlier hinton run also got a recorded walkthrough video (youtu.be/dKqA962fIwQ) plus a new experiment kickoff for the Schmidhuber set (PR #2, youtu.be/w_ybQjbIYYA).

The wikitext topic was the busiest of the week at 67 messages. Andy landed on a two-track split for it: a systems track (Modal, Colab, or Lambda on an A100 or pinned GPU, NVML energy, a real leaderboard) and a local or dev track (deterministic fixture, local evaluator). Armins pushed back with several questions about what the track separation means in practice, whether MPS energy work cross-applies to H100s, and how small the training-data fraction can get before the benchmark stops being meaningful. Yaroslav's framing was that energy-efficient learning is a less data-sensitive task than data-efficient learning since the goal is matching the baseline with less energy, and that next-token prediction and matmul-on-a-grid are connected through the underlying learning problem. Dataset-ladder idea floated by Yaroslav: Shakespeare to TinyStories to Wikitext-103 to FineWeb, with TinyStories citing arXiv:2309.06979. Reference points raised included Keller Jordan's modded-nanogpt, Karpathy's nanoGPT and an llm.c issue, and Yaroslav's own m5-gemm matmul work on Apple silicon that got within 80 percent of advertised capacity. WebGPU came up as a shared todo. Yaroslav's repro attempt stalled: Lambda was out of GPUs and the Modal run got stuck downloading 500MB of wikitext-103 from a throttled HF endpoint.

On challenge #3, energy-efficient sparse parity, Yaroslav added a 50 percent accuracy target (README medium-50 section) built on Sung Jae's packed solution, and added anti-cheating hardening via Gemini that swapped in secrets.SystemRandom. He noted that after more poking this could become a good candidate for a global competition. Related work continued on the matmul side, where Yaroslav showed the shift in read distance going from naive to tiled matmul (access_distance_plots).

A recurring theme was how to make the human role legible. Yaroslav suggested the human acts more like a grant review board, checking what has been done and deciding whether to extend funding, and asked that each PR carry a short human-written section on motivation or lessons learned, since human communication stays sparse while agents produce a lot. Anastasiia created an agent-harness repo. Andy mentioned he is headed to RustWeek on the 18th and will try to meet the rust-gpu creator. Yaroslav also spun up a schmidhuber-problems stub set and pointed at sutro-problems/matmul as a more realistic metric to port. PRs moving through the week included sutro-problems #8, #9 (language-modelling task from Armins), and #14 (Modal, also Armins, occasionally stuck in Modal's queue), plus hinton-problems #53 from Andy. Yaroslav noted Claude Code's 5-hour limits got doubled.

Open question carried forward: Armins is still probing whether energy-efficient MPS training has value for its own sake or only insofar as the learnings cross-apply to GPU clusters.

Sources