predictable-stereo

Schmidhuber, J., & Prelinger, D. (1993). Discovering predictable classifications. Neural Computation 5(4):625–635. TR CU-CS-626-92, University of Colorado at Boulder. paper page | companion: Becker, S., & Hinton, G. E. (1992). Self-organising neural network that discovers surfaces in random-dot stereograms. Nature 355:161–163 (the IMAX paper).

Problem

Predictability maximization (the dual of predictability minimization). Two networks each see one view of the same scene; their job is to produce scalar codes that maximally agree. The only thing the two views actually share is a hidden binary “depth” variable; everything else is view-specific distractor noise. So the only way to make the two codes agree is to extract that hidden variable.

We use the Becker-Hinton 1992 IMAX objective (their equation 4):

I(y_L; y_R) = 0.5 * log( var(y_L + y_R) / var(y_L - y_R) )

which under the Gaussian assumption equals the mutual information between the two scalar outputs. We minimize the negative.

Synthetic binary stereo

Each sample has a hidden depth bit z_i ∈ {-1, +1} and two views, each of dimension d_shared + d_view = 16:

The two templates are random {-1, +1} vectors of length 8, fixed across the dataset, different between the two views. From a single view, the shared dims and the distractor dims look statistically identical (both uniform {-1, +1} marginally) — without the partner view, you cannot tell which dims to attend to. The pred-max objective is what supplies the inductive bias.

The Schmidhuber-Prelinger 1993 paper itself works with binary classifications discovered from co-occurring “contexts.” We use the Becker-Hinton-style synthetic stereo input that is the canonical concrete example of the same predictability-max idea, since the original 1993 TR is not retrievable in detail. See §Deviations.

Files

Running

# Reproduce the headline result.
python3 predictable_stereo.py --seed 0 --n-epochs 200
# (~0.1 s on an M-series laptop CPU; see §Results.)

# Negative control: same training, no shared depth between L and R.
python3 predictable_stereo.py --seed 0 --n-epochs 200 --shuffled

# Multi-seed sweep (real stereo).
python3 predictable_stereo.py --seeds 0,1,2,3,4,5,6,7 --n-epochs 200

# Smoke test (~0.02 s).
python3 predictable_stereo.py --seed 0 --quick

# Regenerate visualizations and GIF.
python3 visualize_predictable_stereo.py --seed 0
python3 make_predictable_stereo_gif.py --seed 0 --n-epochs 200 --fps 6

Results

Configuration (seed 0, headline run):

Headline (seed 0):

Multi-seed sweep (8 seeds, real stereo):

Mean held-out recovery 0.997 (min 0.994, max 1.000, 8/8 seeds). Seed 3 plateaus at a smaller IMAX value (I ~ 3.46 nats vs ~7.6 for the others) but still recovers the hidden bit at 0.998 — the network found a working detector that did not push the variances all the way to the eps floor.

Negative-control sweep (4 seeds, --shuffled: right view’s depth is a permutation of the left view’s, so there is no shared variable):

Mean held-out recovery on the shuffled control: 0.513 (chance level), even though the IMAX loss happily drives its own ratio down — see §Open questions for what the network finds in this case.

Headline: two-network IMAX-style predictability maximization recovers the shared binary depth variable on held-out synthetic stereo at 0.997 average accuracy across 8 seeds, vs 0.513 chance accuracy on the shuffled negative control.

Visualizations

Deviations from the original

The Schmidhuber-Prelinger 1993 Neural Computation paper is partially retrievable; the canonical secondary description of the predictability-max idea is the Becker-Hinton 1992 Nature paper, which sketches the IMAX objective and the random-dot-stereogram task. Each deviation below has a one-line reason.

Open questions / next experiments

1. Output-collapse on the shuffled control. On --shuffled the IMAX loss still drives down past -5 nats and the binary agreement reaches 0.999 even though there is no shared variable. The networks find a pair of functions that output almost the same constant on almost all inputs, which is a var → 0 degenerate optimum. Held-out recovery stays at chance, which is the honest signal. The fix is the variance-regularizer from Becker 1996 (penalize (var(y) - target)^2) or the entropy-regularizer from Schmidhuber’s later work. Worth adding as a v1.5 follow-up.
2. Discrete classifications. The 1993 Neural Computation paper is specifically about discovering classifications, i.e. discrete codes, not real-valued ones. A natural follow-up is to train a softmax head with the Schmidhuber-Prelinger discrete predictability score (cross-entropy of one network’s classification predicted from the other’s) instead of IMAX, and compare convergence speed and robustness. The continuous relaxation we use is in spirit the same idea but a different optimization surface.
3. More than one shared variable. Multi-bit shared structure (k>1 independent hidden bits) requires either k independent (y_L, y_R) heads trained with a decorrelation penalty, or a vector-valued IMAX. The first is the “multiple modules” setup of the 1993 paper. Both are straightforward extensions of this code.
4. Real random-dot stereograms. The Becker-Hinton 1992 Nature task is the canonical demonstration. Reconstructing 5x5 binary patches with parameterised disparity, training the same IMAX objective on the same architecture, and reporting disparity-discrimination accuracy would close the gap to the original Becker-Hinton experiment. It would also check whether the convolutional / patch-shared-weight version of the IMAX objective discovers the same disparity sensitivity.
5. Mode-counting interpretation. The trained network ends up with I(y_L; y_R) ~ 7.6 nats. log(2) ~ 0.69 nats per bit, so naively this reads as ~11 bits of shared information — way more than the one bit actually present in z. The IMAX MI estimate is in fact a Gaussian surrogate that overestimates when the outputs are sharp (saturated tanh). Replacing the IMAX surrogate with a binned histogram MI estimator would give a more honest readout. Interesting micro-experiment.
6. v2 instrumentation. Under ByteDMD, the IMAX update has a particular data-movement signature: each step computes var(y_L + y_R) and var(y_L - y_R) over the full batch, then back-propagates a small per-sample correction. The two networks’ forward+backward passes are completely independent given the corrections (an “outer product” form), which makes this a cheap pipeline for data-movement-conscious training. Worth measuring.

This stub is part of Wave 5 (predictability min/max + unsupervised features) of the schmidhuber-problems catalog. See SPEC issue #1 for the catalog-wide contract.