9.6 Neural Networks as High-Dimensional Mandelbrot Sets

Training a neural network is, in a precise sense, computing a Mandelbrot set in a space of hundreds of thousands of dimensions.

The weight space is the high-dimensional analog of the complex plane. The loss landscape is the iteration, each training step applies the rule, adjusts the weights, and checks whether the system is converging or diverging. The trained network is a point that didn’t escape, a stable attractor in weight space.

The boundary between convergence and divergence in weight space is where the interesting behavior lives. Models near that boundary generalize well, they are poised between memorizing training data (stable interior) and failing to learn anything (escape to infinity). The best models sit on the edge.

The bulbs of the Mandelbrot set correspond to the basins of attraction, the regions of weight space that converge to good solutions. Larger bulbs correspond to more robust solutions, reachable from a wider range of starting points. The branching structure of the boundary corresponds to the fine structure of generalization, the increasingly subtle distinctions a deep model can make.

This is not a metaphor. It is a structural homology between two iterative dynamical systems, one in two dimensions and one in hundreds of thousands.