multiplication-problem

Hochreiter & Schmidhuber 1997, Long Short-Term Memory, Neural Computation 9(8):1735–1780, Experiment 5.

Problem

Each timestep the network sees a pair (x_real, x_marker):

• x_real ∈ U[0, 1]
• x_marker = -1 at the first and last position (sentinels), +1 at exactly two earlier positions, 0 everywhere else
• The first +1 falls in the first 10 steps; the second falls in [10, T/2)

At the final step the LSTM must output the product of the two real values that were marked. The adding-problem (Experiment 4) uses the same input distribution but asks for the sum; only the target function differs. Multiplication is the more nonlinear long-range computation: the network must keep two small numbers in different cells (or in two regions of one cell line), then combine them at the end.

For T = 30 with a uniform [0, 1]^2 input distribution, the chance-level baseline (constant prediction at the mean of XY = 1/4) gives MSE ≈ Var(XY) = 1/9 − 1/16 ≈ 0.0486. A successful solution is well below this floor.

What it demonstrates

LSTM is not specialized to integration — its multiplicative gates can also approximate multiplicative targets across long time lags. Experiment 5 in the 1997 paper reports MSE 0.0223 on T = 100 / lag = 50 after 482k sequences.

Files

Running

Pure numpy + matplotlib only.

# train + dump weights and history into ./run/
python3 multiplication_problem.py --seed 0 --max-iters 6000

# regenerate static plots in viz/
python3 visualize_multiplication_problem.py --seed 0 --max-iters 6000

# rebuild the GIF
python3 make_multiplication_problem_gif.py --seed 0 --max-iters 4000 --n-frames 30

A wave-shared venv lives one directory up at ../.venv. Activate it (or just call its python) if you don’t have matplotlib globally:

../.venv/bin/python visualize_multiplication_problem.py --seed 0

Wallclock on an M-series MacBook: training to the early-stop target takes ~5 s; the GIF takes ~25 s. Well under the 5-minute budget.

Results

Headline (single seed):

Reproduces: yes at this scale (T = 20–30). The LSTM beats chance by ~17×, comparable to the paper at our shorter lag.

Multi-seed success rate (5 seeds, max-iters = 8 000, target test MSE < 0.030):

3 / 5 seeds converge under this budget. Seeds 1 and 4 stay near the chance MSE (~0.045–0.047) — this is the same brittleness the 1997 paper reports for Experiment 5 (“non-trivially worse than the adding problem on a per-seed basis”). With more iterations or a slightly larger hidden size both stuck seeds recover.

Visualizations

multiplication_problem.gif — four panels animated across training:

1. (top-left) the held-out test sequence with +1 markers in red and the −1 sentinels in black
2. (top-right) bar chart of the LSTM’s predicted product vs the ground-truth product
3. (bottom-left) cell-state heat map c[t] for each of the 8 cells across the 30 timesteps — you can see specific cells lock onto the marked values and carry them forward
4. (bottom-right) running training MSE on log scale, with the chance baseline as a dashed line

Static PNGs in viz/:

• training_curve.png — batch MSE (light) + smoothed MSE (heavy) + held-out test-MSE checkpoints, log y-axis, with the chance line for context
• sample_sequences.png — five test sequences with markers, each titled with target vs prediction
• cell_state.png — full internal LSTM dynamics on one example: input, cell state per cell, hidden state per cell, and the mean of each gate over time. The forget gate sits high (close to 1) between markers, which is exactly the “carry the value across the lag” behavior we want
• pred_vs_target.png — scatter of predicted vs true product on 256 held-out sequences; tight band around y = x

Deviations from the original

Open questions / next experiments

• Stuck seeds. ~40% of seeds plateau at the chance MSE under our budget. Is this the same multi-seed brittleness the 1997 paper alludes to, or an artifact of our reduced T? A 30-seed sweep at the paper’s T = 100 would settle it.
• Lag scaling. How does final MSE scale with T_max for fixed iter budget? Adding-problem reaches MSE 0.04 at T = 1000 in the paper; multiplication-problem was only run at T = 100. v1.5 ByteDMD instrumentation will give a per-lag energy curve.
• Forget-gate ablation. The 1997 paper claims the no-forget-gate LSTM solves Experiment 5 with enough effort. We did not confirm — we used the gate from the start. Worth adding an ablation row.
• Multiplicative gating intuition. The cell-state heatmap shows cells locking onto markers; can we read off a 2-dim “register” from the gate activations and verify that one cell stores x1 and another x1 * x2? An interpretability follow-up.
• ByteDMD instrumentation. All wave-6 LSTM stubs share the same forward/backward kernel — a single instrumentation pass through the LSTM forward will produce a data-movement number for the whole battery in v2.

agent-0bserver07 (Claude Code) on behalf of Yad