Speaker
Description
Wide fully connected neural networks admit a remarkably tractable probabilistic description at large width. At fixed depth, their outputs converge to Gaussian processes, while the empirical covariance kernels of successive layers evolve according to a deterministic nonlinear recursion. Interpreting depth as a discrete time variable, this places the standard neural-network Gaussian process (NNGP) theory naturally in the language of dynamical systems.
I will discuss how large-deviation theory enriches this picture. Rather than describing only the typical kernel trajectory, a large-deviation principle assigns a cost to atypical transitions between successive layer kernels. This leads to a variational formulation on paths: the usual NNGP dynamics appears as the zero-cost trajectory, while atypical outputs and representations are obtained by minimizing an action subject to suitable endpoint constraints. In this sense, large deviations provide a least-action counterpart to the forward NNGP recursion.
This perspective suggests connections with stability and criticality of the kernel dynamics, including the usual edge-of-chaos picture, as well as with Bellman principles, optimal trajectories and rare transitions in deep networks. I will illustrate these ideas with examples for fully connected networks and discuss how conditioning or statistical losses naturally enter the same framework as additional variational costs.
The talk is partly based on joint works with Claudio Macci, Barbara Pacchiarotti, Katerina Papagiannouli, Giovanni Luca Torrisi and Giuseppe Pio Zito.