sutro-problems

View this project on GitHub ↗

MNIST end-to-end on an A100

Architecture + optimizer + kernel codesign

Screenshot 2026-09-29 at 5 14 36 PM

API

import mnist

def my_method(train_x, train_y, test_x):
    # train_x (10000, 60) float32, train_y (10000,) int64 in 0..9, test_x (10000, 60) float32, on the GPU
    ...
    return labels  # (10000,) integer tensor on the GPU

if __name__ == "__main__":
    ms = mnist.score(my_method, difficulty=1)  # 1 (loosest) to 5; ms per call, or raises mnist.Disqualified
python example.py                                            # on your own GPU
python run_modal.py example.py:mlp --difficulty 1 --runs 3   # official: three Modal A100-80GB runs, median

The rules are in mnist.py’s docstring.

Each row is a separate submission by the listed entrant; source attribution and verification scope are in its report. Rows with one verification run are provisional. ★ identifies an established record. Non-record submissions are welcome. Research with failed or incomplete qualification checks can be included with all outcomes documented in the research submissions below; its inclusion does not qualify it for a difficulty table.

Difficulty = 1 (error ≤ 5.40%)

Date mJ ms Submission Entrant Description
2026-09-25 29,662 700.1 py, report @yaroslavvb example.py baseline: MLP 60-1024-1024-10, 400 steps
2026-09-26 2,648 191.3 py, report @yaroslavvb MLP 60-1024-1024-10, 100 steps
2026-09-26 2,131 61.7 py, report @yaroslavvb MLP 60-256-256-10, 200 steps in a CUDA graph
2026-09-30 3,634 351.0 py, report @yaroslavvb MLP, 200 steps, TF32 and fused AdamW; one sandboxed run
2026-10-08 530 6.7 py, report @sjbaebae Two 60-256-256-10 MLPs, 30 updates, three-neighbor voting; median of three sandboxed Modal runs
2026-10-12 60 1.6 py, report @spartanhaden Nyström RBF kernel ridge in a Fisher metric, 480 class-wise k-means landmarks, one Cholesky solve; no backprop; median of three independent reviewer sandboxed Modal runs ★ best

Difficulty = 2 (error ≤ 3.40%)

Date mJ ms Submission Entrant Description
2026-09-26 200,870 927.3 py, report @yaroslavvb 16 MLPs 60-1024-1024-10, 400 steps
2026-09-26 14,436 252.1 py, report @yaroslavvb 4 MLPs 60-256-256-10, 800 steps in a CUDA graph
2026-09-29 8,071 38.3 py, report @islamborghini Non-neural RBF kernel ridge, one learned metric update, 16-step Nyström-preconditioned CG
2026-09-30 132,635 761.9 py, report @yaroslavvb 8 MLPs, 500 steps, EMA 0.992; one sandboxed run
2026-10-08 1,333 14.6 py, report @sjbaebae Two 60-256-256-10 MLPs, 120 updates, three-neighbor voting; median of three sandboxed Modal runs
2026-10-12 603 4.6 py, report @spartanhaden RBF kernel ridge with one learned (AGOP) metric, 5 Nyström-preconditioned CG steps; no backprop; median of three independent reviewer sandboxed Modal runs
2026-10-19 208 3.2 py, report @spartanhaden Fisher-metric RBF ridge on a compressed weighted training set, triangular FITC-style preconditioning and extrapolated PCG; ports sigkillme0’s D3 method; median of three independent reviewer sandboxed Modal runs
2026-10-22 172 2.9 py, report @spartanhaden Fisher-PCG D2 with a standard-form Woodbury preconditioner, FP16 products, side-stream class moments and batched small factors; learner constants unchanged; median of three independent reviewer sandboxed Modal runs ★ best

Difficulty = 3 (error ≤ 2.70%)

Date mJ ms Submission Entrant Description
2026-09-28 1,159,909 15,818.8 py, report @SethTS Ladder 60-1000-500-250-250-250-10, 5,000 steps in a CUDA graph
2026-09-28 676,850 6,556.9 py, report @SethTS Ladder, 6,500 steps, fused Triton kernels in a CUDA graph
2026-09-29 259,847 1,895.4 py, report @SethTS Ladder, batch 1,000, 1,200 steps, TF32, fused Triton kernels in a CUDA graph
2026-09-29 201,792 1,443.3 py, report @SethTS Ladder, batch 1,000, 900 steps, labelled rows sampled by loss, TF32, fused Triton kernels in a CUDA graph
2026-09-30 253,761 1,732.2 py, report @yaroslavvb Ladder, 1,100 steps; 8.3% fewer updates; one sandboxed run
2026-10-05 151,404 1,157.1 py, report @islamborghini Sampled Ladder, batch 1,000, 800 steps, TF32; tuned forward and backward Triton kernels; median of three sandboxed runs
2026-10-08 98,226 416.1 py, report @sjbaebae Eight 60-256-256-256-10 RMSNorm/ReLU MLPs, 1,250 updates, five-neighbor voting and three-step query graph propagation; median of three sandboxed Modal runs
2026-10-12 8,839 51.8 py, report @sjbaebae Spherical direction/radius RBF, two tangent-gradient metric updates; derived from PR #116; median of three independent reviewer sandboxed Modal runs
2026-10-12 2,362 18.1 py, report @spartanhaden RBF kernel ridge, two AGOP metric updates, HKNN local-hull correction on the 512 lowest-margin rows; no backprop; median of three independent reviewer sandboxed Modal runs
2026-10-14 7,899 44.0 py, report @islamborghini RBF kernel ridge, two analytic metric updates, CUDA-graph Nyström-preconditioned CG and local class geometry; median of three independent reviewer sandboxed Modal runs
2026-10-18 320 5.2 py, report @sigkillme0 Learned-metric RBF kernel with fused Triton PCG, compressed kernel tiles, and local-hull correction; builds on Haden’s RFM/HKNN learner; median of three independent reviewer sandboxed Modal runs
2026-10-19 639 4.6 py, report @sjbaebae Fisher-PCG with 6,144 hard rows, eight PCG steps and native FP32 local-hull CG; builds on Haden, sigkillme0 and Islam’s kernel work; median of three independent reviewer sandboxed Modal runs
2026-10-22 359 3.5 py, report @spartanhaden Plain-PyTorch Fisher-PCG with a rank-48 metric, natural-gradient step, Woodbury preconditioner and HKNN on compacted low-margin rows; median of three independent reviewer sandboxed Modal runs ★ best

Difficulty = 4 (error ≤ 2.30%)

Date mJ ms Submission Entrant Description
2026-09-28 1,143,255 11,094.6 py, report @SethTS Ladder, 11,000 steps, fused Triton kernels in a CUDA graph
2026-09-29 547,382 3,784.2 py, report @SethTS Ladder, batch 1,000, 2,400 steps, TF32, fused Triton kernels in a CUDA graph
2026-09-29 400,682 2,882.5 py, report @SethTS Ladder, batch 1,000, 1,800 steps, labelled rows sampled by loss, TF32, fused Triton kernels in a CUDA graph
2026-09-30 460,763 3,481.3 py, report @yaroslavvb Ladder, 2,200 steps; 8.3% fewer updates; one sandboxed run
2026-10-12 63,113 580.1 py, report @spartanhaden Ladder MLP 60-1024-512-256-10 on AGOP-metric inputs, 560 Adam steps, kernel-ridge teacher; median of three independent reviewer sandboxed Modal runs
2026-10-18 42,617 219.2 py, report @islamborghini Five RBF classifiers with three analytic AGOP metric updates each, Nyström-preconditioned CG, transductive refits and HKNN affine-hull correction; no backprop; median of three independent reviewer sandboxed Modal runs
2026-10-19 14,754 97.0 py, report @spartanhaden Two RFM teachers distilled into three tangent-jittered MLP students with cross-teaching and HKNN; one full-call CUDA graph; median of three independent reviewer sandboxed Modal runs
2026-10-19 12,845 78.3 py, report @sjbaebae Haden’s tangent hybrid with 250 Adam updates and cross-teaching at 80/170 steps; teacher, students and readout unchanged; median of three independent reviewer sandboxed Modal runs
2026-10-22 6,664 38.7 py, report @spartanhaden Two Fisher-PCG teachers distilled into three tangent-jittered MLP students; 220 updates with a fused Triton student step and HKNN; median of three independent reviewer sandboxed Modal runs
2026-10-25 3,347 25.3 py, report @sjbaebae Fisher-PCG teacher distilled into three width-128 tangent students, split-four gradients and rolled Triton row loops; retains Haden and upstream kernel credits; median of three independent reviewer sandboxed Modal runs
2026-11-01 2,259 22.1 py, report @spartanhaden Fisher-PCG teacher with three width-128 tangent students, class-wise neighbors, without-replacement row walks, 1,280 labelled and 256 unlabelled rows, and 220 fused Triton updates; retains #140 and upstream credits; median of three independent reviewer sandboxed Modal runs ★ best

Difficulty = 5 (error ≤ 1.90%)

Date mJ ms Submission Entrant Description
2026-09-28 4,410,730 36,201.2 py, report @SethTS Ladder, 36,000 steps, fused Triton kernels in a CUDA graph
2026-09-29 2,059,576 14,130.6 py, report @SethTS Ladder, batch 1,000, 9,000 steps, TF32, fused Triton kernels in a CUDA graph
2026-09-30 1,824,760 13,264.3 py, report @yaroslavvb Ladder, 8,400 steps; 6.7% fewer updates; one sandboxed run
2026-10-12 638,234 3,335.4 py, report @likelytobelaura Eight RMSNorm/GELU MLPs, Muon/AdamW, 2,500 updates, local tangent jitter, disjoint-pool cross-teaching and graph readout; median of three independent reviewer sandboxed Modal runs
2026-10-12 226,352 1,818.1 py, report @spartanhaden Ladder MLP 60-1024-512-256-10 on AGOP-metric inputs, 2,300 Adam steps, kernel-ridge teacher; median of three independent reviewer sandboxed Modal runs
2026-10-14 343,152 1,915.4 py, report @likelytobelaura Eight width-256 RMSNorm/ReLU MLPs, Muon, tangent jitter, cross-teaching and graph readout; median of three independent reviewer sandboxed Modal runs
2026-10-18 154,289 897.5 py, report @islamborghini Haden’s kernel-taught Ladder tuned to batch 2,000 and 800 steps; architecture, teacher and optimizer unchanged; median of three independent reviewer sandboxed Modal runs
2026-10-19 107,886 633.5 py, report @spartanhaden Kernel-taught Ladder with tangent jitter, hard-query sampling and fused Triton combinator backward; 600 Adam steps of batch 2,500; median of three independent reviewer sandboxed Modal runs
2026-10-19 77,090 492.8 py, report @sjbaebae Haden’s kernel-taught Ladder narrowed to 768-512-256, 480 native fused Adam updates and batch 2,500; teacher and branchless guard retained; median of three independent reviewer sandboxed Modal runs
2026-10-22 67,813 406.9 py, report @spartanhaden Kernel-taught width-768 Ladder, 450 Adam updates, labelled-row top-down targets, per-call noise bank and hidden-space HKNN; median of three independent reviewer sandboxed Modal runs
2026-10-25 40,390 273.9 py, report @sjbaebae Width-480 label-top-down Ladder, 315 updates, per-call transpose/near-identity initialization, population prediction statistics, parallel gradient reductions and decoder/hull readout; retains Ladder and kernel lineage credits; median of three independent reviewer sandboxed Modal runs
2026-11-01 30,157 226.9 py, report @spartanhaden Width-480 Ladder with decoupled weight decay on eight encoder/decoder matrices and 240 AdamW updates; retains #140 and upstream Ladder/kernel credits; median of three independent reviewer sandboxed Modal runs ★ best

Research submissions

These non-record experiments are included with their original runtime code and author evidence. The outcomes below come from the retained independent reviewer runs on those frozen sources. Each report separates author results from reviewer results and preserves failed and incomplete runs. Research inclusion does not qualify a result for a difficulty table or establish a record.

Experiment Reviewer outcome Remaining limitations
D4 cold-compilation repair (#114) 1 of 3 passed Two complete runs missed by 64 and 4 labels; full source lineage remains unsettled.
D5 tangent-jitter ensemble (#119) 1 of 3 passed; two incomplete Later foreign-data warm-ups timed out after seven timed calls in each incomplete run.
D5 width-128 Muon ensemble (#122) 1 of 3 passed Two complete runs missed by 28 and 74 labels; both original author groups remain separately documented.
Neural learner submitted for D4 (#126) 1 of 3 passed Two complete runs missed by 26 and 20 labels; author timing records identify a different, unsupplied source.
Ladder variant submitted for D5 (#127) 1 of 3 passed Two complete runs missed by 116 and 57 labels; remaining source-lineage limits are documented.
D2 reduced Fisher-PCG (#133) 2 of 3 passed The third complete run was one prediction below the cutoff.