BEGIN:VCALENDAR VERSION:2.0 PRODID:-//linuxsoftware.nz//NONSGML Joyous v1.4//EN BEGIN:VEVENT SUMMARY:Learning Dynamics of Neural Networks DTSTART:20250702T163000Z DTEND:20250702T180000Z DTSTAMP:20260702T173451Z UID:5d763664-7746-44b3-a71c-82d3202da980 SEQUENCE:2 CREATED:20250626T133254Z DESCRIPTION:This thesis investigates the learning dynamics of neural netwo rks through a combination of statistical mechanical and dynamical systems tools\, within the controlled setting of the teacher–student framework. We use this setup by introducing a minimal student model trained to reprod uce the outputs of a fixed teacher network via Stochastic Gradient Descent on a mean-squared error loss. By analyzing the Hessian of the loss functi on\, we characterize the local curvature of the landscape at and near opti mal points\, revealing how overparameterization and activation function ch oice shape the spectrum of eigenvalues and\, hence\, the rates of converge nce. To probe transient chaotic behavior during training\, we compute Loca l Lyapunov Spectra and observe that\, even in low-dimensional teacher-stud ent tasks\, there is local chaoticity that can theoretically lead to expon entially diverging parameter trajectories before they settle into stable m inima or flat manifolds. Principal Component Analysis of these trajectorie s further uncovers a marked reduction in effective dimensionality over the course of training\, with the majority of variance confined to leading mo des that coincide with directions of minimal curvature in the Hessian. Fin ally\, when extending our framework to a network learning from a dataset o n the MNIST classification task\, we find that entropy measures derived fr om positive Local Lyapunov Exponents do not correlate with generalization performance\, highlighting the need for alternative complexity metrics in realistic\, high-dimensional settings. LAST-MODIFIED:20250626T133304Z LOCATION:Anfiteatro QA1.2\, Piso 1\, Pavilhão de Química\, Campus Alamed a URL:http://df.vps.tecnico.ulisboa.pt/pt/eventos/learning-dynamics-of-neura l-networks/ X-ALT-DESC;FMTTYPE=text/html:

This thesis investi gates the learning dynamics of neural networks through a combination of st atistical mechanical and dynamical systems tools\, within the controlled s etting of the teacher–student framework. We use this setup by introducin g a minimal student model trained to reproduce the outputs of a fixed teac her network via Stochastic Gradient Descent on a mean-squared error loss.< br/>
By analyzing the Hessian of the loss function\, we characterize the local curvature of the landscape at and near optimal points\, revealin g how overparameterization and activation function choice shape the spectr um of eigenvalues and\, hence\, the rates of convergence. To probe transie nt chaotic behavior during training\, we compute Local Lyapunov Spectra an d observe that\, even in low-dimensional teacher-student tasks\, there is local chaoticity that can theoretically lead to exponentially diverging pa rameter trajectories before they settle into stable minima or flat manifol ds.

Principal Component Analysis of these trajectories further u ncovers a marked reduction in effective dimensionality over the course of training\, with the majority of variance confined to leading modes that co incide with directions of minimal curvature in the Hessian.

Fina lly\, when extending our framework to a network learning from a dataset on the MNIST classification task\, we find that entropy measures derived fro m positive Local Lyapunov Exponents do not correlate with generalization p erformance\, highlighting the need for alternative complexity metrics in r ealistic\, high-dimensional settings.

END:VEVENT END:VCALENDAR