Hinton Problems

A reproducible-baseline catalog of the synthetic learning problems that appear in Geoffrey Hinton’s experimental papers from 1981 through 2022 — implemented in pure numpy, runnable on a laptop CPU, with paper-comparison metrics per stub.

Site: https://cybertronai.github.io/hinton-problems/ • Catalog: RESULTS.md • 53 of 53 stubs implemented (PRs #32–#41, all merged 2026-05-03)

Introduction

The field has standardized on backprop by the end of the ’80s, and Hinton gives a sample of problems that were used at the time. In the last 20 years, we have transitioned to GPUs, and the math has changed considerably. Instead of being bottlenecked by arithmetic, the shrinking of transistors means that arithmetic is essentially free, and all of the work comes from data movement. Backprop is inefficient in terms of “commute to compute ratio” because it requires fetching all of the activations for each gradient add.

So a natural experiment would be to redo key experiments of this time with a focus on data movement. The first step is to get a baseline — to establish the list of problems which are famous (made by Hinton), reasonable to implement, and easy to run/reproduce.

— Yaroslav, issue #1 (Sutro Group)

This repository is that baseline. v1 ships 53 implementations covering the lineage from the 4-2-4 encoder (1985) through the shifter (1986), bars (1995), MultiMNIST (2017), Constellations (2019), Ellipse World (2022), and the Forward-Forward suite (2022). Each stub is a self-contained folder with model + train + eval + visualization + animated GIF, all in numpy, all runnable in <5 min per seed on an M-series laptop.

The next step (#45 v2) instruments these 53 baselines with ByteDMD — Yaroslav’s data-movement cost tracer — to measure the actual “commute” each algorithm pays.

What’s here

Pure numpy + matplotlib throughout. Every stub runs on a laptop CPU. Each problem lives in its own folder with <slug>.py (model + train + eval), README.md, make_<slug>_gif.py, visualize_<slug>.py, an animated <slug>.gif, and a viz/ folder of training curves and weight visualizations.

Visual tour

encoder-4-2-4 — Ackley/Hinton/Sejnowski 1985, the worked example. Bipartite RBM, 2-bit code emerges.	spline-images-factorial-vq — Hinton/Zemel 1994, factorial VQ wins 3× over standard 24-VQ baseline.
ellipse-world — Culp/Sabour/Hinton 2022, eGLOM islands form across iterations (5-class, 92.2%).	ff-recurrent-mnist — Hinton 2022, top-down recurrent Forward-Forward.

Catalog

Each table shows the v1 result per stub. Full per-stub metrics (compile-time, GIF size, headline numbers) are in RESULTS.md.

Reproduces? legend: yes = matches paper qualitatively or quantitatively; partial = method works, paper number not fully reached (gap documented in stub README); no = paper claim does not replicate.

1980s — Connectionist foundations

Ackley, Hinton & Sejnowski (1985) — A learning algorithm for Boltzmann machines

Rumelhart, Hinton & Williams (1986) — Learning internal representations by error propagation

Hinton (1986) — Distributed representations of concepts

Hinton & Sejnowski (1986) — Learning and relearning in Boltzmann machines

Plaut & Hinton (1987) — Learning sets of filters using back-propagation

Hinton & Plaut (1987) — Using fast weights to deblur old memories

1990s — Unsupervised learning, mixtures, the Helmholtz machine

Jacobs, Jordan, Nowlan & Hinton (1991) — Adaptive mixtures of local experts

Becker & Hinton (1992) — A self-organizing neural network that discovers surfaces in random-dot stereograms

Nowlan & Hinton (1992) — Simplifying neural networks by soft weight-sharing

Hinton & Zemel (1994) — Autoencoders, MDL and Helmholtz free energy

Zemel & Hinton (1995) — Learning population codes by minimizing description length

Dayan, Hinton, Neal & Zemel (1995) — The Helmholtz machine

Hinton, Dayan, Frey & Neal (1995) — The wake-sleep algorithm

2000s — Products of experts, contrastive divergence, deep belief nets

Hinton (2000) — Training products of experts by minimizing contrastive divergence

Memisevic & Hinton (2007) — Unsupervised learning of image transformations

Sutskever & Hinton (2007) — Multilevel distributed representations for high-dimensional sequences

Sutskever, Hinton & Taylor (2008) — The recurrent temporal RBM

2010s — Capsules, distillation, attention

Hinton, Krizhevsky & Wang (2011) — Transforming auto-encoders

Tang, Salakhutdinov & Hinton (2012) — Deep Lambertian Networks

Sutskever, Martens, Dahl & Hinton (2013) — On the importance of initialization and momentum

Hinton, Vinyals & Dean (2015) — Distilling the knowledge in a neural network

Eslami, Heess, Weber, Tassa, Szepesvari, Kavukcuoglu & Hinton (2016) — Attend, Infer, Repeat

Ba, Hinton, Mnih, Leibo & Ionescu (2016) — Using fast weights to attend to the recent past

Sabour, Frosst & Hinton (2017) — Dynamic routing between capsules

Hinton, Sabour & Frosst (2018) — Matrix capsules with EM routing

Kosiorek, Sabour, Teh & Hinton (2019) — Stacked capsule autoencoders

2020s — Subclass distillation, GLOM, Forward-Forward

Müller, Kornblith & Hinton (2020) — Subclass distillation

Sabour, Tagliasacchi, Yazdani, Hinton & Fleet (2021) — Unsupervised part representation by flow capsules

Culp, Sabour & Hinton (2022) — Testing GLOM’s ability to infer wholes from ambiguous parts

Hinton (2022) — The forward-forward algorithm: some preliminary investigations

Structure

problem-folder/
├── README.md                  source paper, problem, results, deviations
├── <slug>.py                  dataset + model + train + eval
├── visualize_<slug>.py        training curves + weight viz
├── make_<slug>_gif.py         animated GIF
├── <slug>.gif                 committed animation
└── viz/                       committed PNGs

Roadmap

• #45 v2: ByteDMD instrumentation — measure data-movement cost per stub on these baselines (the actual research goal)
• #46 v1.5: paper-scale reruns — close the 25 partial reproductions on Modal/GPU
• See Open questions / next experiments section in each stub README for stub-specific follow-ups

Contributing

Implementations follow the v1 spec:

• Each stub fills in <slug>.py (model + train + eval), an 8-section README.md, make_<slug>_gif.py, visualize_<slug>.py, an animated <slug>.gif, and viz/ PNGs.
• Acceptance: reproduces in <5 min on a laptop; final accuracy with seed in Results table; GIF illustrates problem AND learning dynamics; “Deviations from the original” section honest; at least one open question.
• v1 metrics in PR body: "Paper reports X; we got Y. Reproduces: yes/no." + run wallclock + implementation wallclock.

The v1.5 reruns (#46) and v2 ByteDMD work (#45) welcome contributions.

License

The hinton-problems source and documentation are released into the public domain under the Unlicense.