Barcodes of Loss Landscapes

Topological data analysis of neural-network loss surfaces. An algorithm computes the barcodes of local minima, and experiments show that deeper and wider networks have friendlier landscapes.

The problem

Gradient descent finds good solutions on loss surfaces that are wildly non-convex, and nobody fully understands why. To study the question you first need a way to measure how bad a landscape’s local minima really are.

The idea

Borrow from topology. Each local minimum is paired with the saddle point at which its basin merges into a deeper one. The height difference between the two is a topological invariant: the penalty an optimiser must pay to escape that minimum. The collection of these segments is the barcode of the loss function.

Existing software computed barcodes on grids with cubic worst-case cost and stalled beyond six dimensions. We described an algorithm that works on arbitrarily sampled point clouds, whose principal part runs in O(N log N), and tested it in up to 15 dimensions on as many as 10⁹ points.

What it showed

• The barcodes of local minima sit in a small lower part of the range of the loss.
• Increasing a network’s depth and width lowers those barcodes.

Both observations point the same way: larger networks have landscapes that are easier to optimise, with implications for learning and generalisation.

With Serguei Barannikov, Alexander Korotin, Dmitry Oganesyan and Evgeny Burnaev. Published in Doklady Rossiiskoi Akademii Nauk. Matematika, Informatika, Protsessy Upravleniya, vol. 514 (2023).