MNIST — 14 Networks

MNIST is solved, so getting a high number is not the exercise. The question I wanted answered was which knob actually matters — depth, width, activation, batch size or training duration — and the only way to know is to change one at a time and record all fourteen results, including the bad ones.

Fourteen feedforward networks from no hidden layers up to three, sweeping width (64–128 neurons), activation (ReLU, Tanh), and batch size. SGD with categorical cross-entropy throughout, so the comparison stays clean.

The three-layer network hit essentially 100% training accuracy at 97.4% validation — textbook overfitting, and a useful reminder that the deepest model on the list is not the one to ship. The best model at 97.50% validation also sat at 99.73% training, so even the winner is memorising more than it generalises.

Best configuration is two hidden layers, 128 ReLU into 64 Tanh, at 97.50% validation. The largest single effect by far was not architecture at all: switching from full-batch to mini-batch SGD at batch size 32 was worth roughly 74 percentage points, dwarfing every depth and width change tested.