Back to work

[ sys ]fast-mnist-nnsha f8efce9measured-in-repo

A digit net I taught the CPU to run fast, from scratch.

I wrote the dense linear algebra and backprop for an MNIST multilayer perceptron in C++17 with no BLAS, Eigen, or PyTorch, then hand-vectorized the hot paths and proved the optimizations pay off with a reproducible benchmark suite.

View the repoJump to the benchmarkTry the live demo
[ fast-mnist-nn ], noun
1.a handwritten-digit MLP whose matrix and MLP hot paths are hand-vectorized.
2.one kernel, four arch paths: AVX-512, AVX2, NEON, scalar tail, chosen at compile time.
3.the losses sit next to the wins in the same chart.
pillar
systems ◆
lang
C++17
license
MIT
size
~3,387 LOC
baseline
own scalar build
harness
Google Benchmark
~3.5×

dot-product speedup

256x256, SIMD intrinsics · 64-byte aligned memory · blocked GEMM tiling

80K+ img/s

inference throughput

784-30-10 MLP, classify path (serial SIMD)

AVX-512 · AVX2 · NEON

compile-time SIMD dispatch

one kernel, four arch paths + scalar tail

The benchmark below is the reproducible evidence behind these figures, each result cited to sha f8efce9.

01Problem

Train and run a digit net fast on the CPU, with nothing borrowed.

Most from-scratch neural nets stop at correctness: a few nested loops over a matrix, a sigmoid, and a training curve that eventually goes down. The harder problem is making the same math fast on a real CPU without reaching for a library that already solved it. I set the constraint up front: no BLAS, no Eigen, no PyTorch. The dense linear algebra and the backprop are mine to write, and the speed is mine to earn.

That turns one project into two questions. Does the network learn MNIST correctly, and does the hand-written math actually run fast enough to matter? The second question only counts if I can prove it, so the benchmark harness is part of the project, not an afterthought.

02Approach

Three moves: align the memory, block the multiply, fuse the step.

Speed here comes from three decisions that compound, each one a deliberate piece of the hot path rather than a compiler flag I flipped on and hoped.

  1. i

    Align the memory.

    Every Matrix rounds its row stride up to a 64-byte boundary and allocates through posix_memalign, so every row starts aligned and the SIMD inner loop uses aligned loads instead of the slower unaligned path.

    src/Matrix.cpp:34
  2. ii

    Block the multiply.

    The multiply walks the contraction dimension in 128-element tiles to keep hot data in cache, and transposes the right operand once so the inner dot reads contiguous memory, then dispatches at compile time to AVX-512, AVX2, NEON, or a scalar tail, with FMA when the target advertises it.

    src/Matrix.cpp:333
  3. iii

    Fuse the step.

    The forward step fuses matrix-vector multiply, bias add, and sigmoid into one pass per weight row. The whole learn step runs forward, backprop, and the SGD update over function-local static scratch buffers, so the training hot path does zero per-step allocation.

    src/NeuralNet.cpp:185

03Architecture

One data path from aligned bytes to a trained step.

The figure follows a single matrix through the optimized path. Aligned rows feed the blocked multiply, the inner dot resolves to one SIMD kernel at compile time, and the scalar result lands in the fused network step. The accent marks only the traversed path. Everything else is structure.

Matrix64B-aligned · padded ldpad → next 64Bblocked DGEMM64×64 tiles · K tiled 128RHS transposed once → contiguous inner loopdotVecSum#if MATRIX_USE_... (compile-time)AVX-5128× f64 + FMAAVX24× f64NEON2× f64scalartail / fallbackfused MLP stepgemv + bias + sigmoid in one pass · static scratch · no per-step allocW·a+ biasσ( )δ outputWᵀ·δSGD update
Scroll the figure sideways to read it. Boxes-and-pointers view of src/Matrix.cpp and src/NeuralNet.cpp at sha f8efce9. The highlighted SIMD lane (AVX-512) is illustrative. The real lane is chosen by the target at compile time.

The library is layered, not a single blob.

Each layer consumes the one beneath it. The CLI drives training and evaluation over a PGM parser. The network owns the fused kernels. The matrix owns aligned storage and the SIMD dispatch. The harness turns benchmark JSON into the committed CSV and charts. About 3,387 lines hold the whole thing.

src layout · top consumes the layer beneath it

  1. apps/fast_mnist_cli.cpptraining + eval driver

    P2 PGM parser · binary image cache

  2. NeuralNet.{h,cpp}fused MLP kernels, static scratch

    gemv+bias+sigmoid · output δ · Wᵀ·δ · SGD

  3. Matrix.{h,cpp}dense linear algebra + SIMD dispatch

    aligned alloc · blocked dot · transpose · axpy

  4. benchmarks/ · tools/run_benchmarks.pyreproducible measurement harness

    Google Benchmark JSON → CSV → SVG

04Tradeoffs · road not taken

Two decisions the data forced.

The data forced two decisions, and neither is clean.

OpenMP only on big matrix ops, never on the network

bench_summary.csv

Threading the small operations made them slower, not faster. With OpenMP on, axpy/128 runs about 6.9× slower and transpose/128 about 4.3× slower, because the fork-join overhead dwarfs the work. So I gate the pragmas behind a size threshold and leave the network's learn and classify paths as serial SIMD. The near-flat learn throughput across all three configs confirms OpenMP never touches them.

Static scratch buffers buy speed, cost thread-safety

src/NeuralNet.cpp

The training and inference paths reuse function-local static buffers to kill per-step allocation. Those statics are shared across every NeuralNet instance, so the win comes at the price of multi-instance and multi-thread correctness. For a single trainer on one thread it is the right call. The reentrancy caveat is in the header.

05Benchmark · vs scalar baseline

Wins and losses, side by side.

The named baseline is the project's single-threaded scalar build (OpenMP off, no -march=native), run side by side with the native and openmp+native configs in the same Google Benchmark harness. There is no BLAS or Eigen comparison, since the whole point was to write the math myself. On the fully optimized matrix path that lands at ~3.5× on the large dot product, the single biggest per-op win. The committed CSV below is the reproducible evidence underneath it.

openmp+native vs scalar baseline · speedup (→ faster) · lower ns/op is better1.0× baseline1×2×3×4×dot 256×2564,835,360 → 1,379,835 ns/op3.5× fasteraxpy 1024230,626 → 114,910 ns/op2.0× fastertranspose 1024978,383 → 502,426 ns/op1.95× fastertranspose 1285,441 → 23,662 ns/op · loss4.3× sloweraxpy 1283,486 → 23,917 ns/op · loss6.9× slower
Scroll the chart sideways to read it. Every value is read from the committed docs/benchmarks/bench_summary.csv at sha f8efce9. Single-repetition runs. Lower ns/op is better. Raw benchmark JSON ↗

The large dot product carries the win: ~3.5× faster at 256×256, with transpose and axpy near 2× at size 1024. The two faint bars below the line are real, not rounding errors. Parallelism has a crossover point, and below it the overhead wins.

Speed alone says nothing about whether the net is right. The int8 model deployed to the browser demo, not the C++ CLI these benchmarks measure, holds 98.85% clean accuracy and 97.62% on an augmented set built to mimic messy handwriting, measured in the accuracy write-up.

[ sys ]fast-mnist-nn

Draw a digit, watch the net decide.

Draw a digit and the network classifies it live in your browser, running the same forward pass as the trained model. The deployed model holds 98.85% accuracy on clean MNIST digits and 97.62% on an augmented set built to mimic messy handwriting, detailed in the accuracy write-up.

preview

Accuracy figures are for the deployed int8 browser model (public/models/mnist-mlp.json), not the C++ training and inference CLI.

Read the source.

The SIMD dispatch, the blocked multiply, the fused kernels, and the benchmark harness are all in the repo, pinned at the commit every number on this page is measured from. Cross-platform CI runs the Catch2 suite on Linux, macOS, and Windows.

View repositoryRead the write-up →← All case studies