BEGIN:VCALENDAR VERSION:2.0 PRODID:-//linuxsoftware.nz//NONSGML Joyous v1.4//EN BEGIN:VEVENT SUMMARY:Learning Dynamics of Neural Networks DTSTART:20250702T163000Z DTEND:20250702T180000Z DTSTAMP:20260725T054457Z UID:5d763664-7746-44b3-a71c-82d3202da980 SEQUENCE:1 CREATED:20250626T133245Z DESCRIPTION: This thesis investigates the learning dynamics of neural netw orks through a combination of statistical mechanical and dynamical systems tools\, within the controlled setting of the teacher–student framework. We use this setup by introducing a minimal student model trained to repro duce the outputs of a fixed teacher network via Stochastic Gradient Descen t on a mean-squared error loss. By analyzing the Hessian of the loss funct ion\, we characterize the local curvature of the landscape at and near opt imal points\, revealing how overparameterization and activation function c hoice shape the spectrum of eigenvalues and\, hence\, the rates of converg ence. To probe transient chaotic behavior during training\, we compute Loc al Lyapunov Spectra and observe that\, even in low-dimensional teacher-stu dent tasks\, there is local chaoticity that can theoretically lead to expo nentially diverging parameter trajectories before they settle into stable minima or flat manifolds. Principal Component Analysis of these trajectori es further uncovers a marked reduction in effective dimensionality over th e course of training\, with the majority of variance confined to leading m odes that coincide with directions of minimal curvature in the Hessian. Fi nally\, when extending our framework to a network learning from a dataset on the MNIST classification task\, we find that entropy measures derived f rom positive Local Lyapunov Exponents do not correlate with generalization performance\, highlighting the need for alternative complexity metrics in realistic\, high-dimensional settings. LAST-MODIFIED:20250626T133245Z LOCATION:Anfiteatro QA1.2\, Piso 1\, Pavilhão de Química\, Campus Alamed a URL:http://df.vps.tecnico.ulisboa.pt/en/events/learning-dynamics-of-neural -networks/ X-ALT-DESC;FMTTYPE=text/html:

This thesis invest igates the learning dynamics of neural networks through a combination of s tatistical mechanical and dynamical systems tools\, within the controlled setting of the teacher–student framework. We use this setup by introduci ng a minimal student model trained to reproduce the outputs of a fixed tea cher network via Stochastic Gradient Descent on a mean-squared error loss.

By analyzing the Hessian of the loss function\, we characterize the local curvature of the landscape at and near optimal points\, reveali ng how overparameterization and activation function choice shape the spect rum of eigenvalues and\, hence\, the rates of convergence. To probe transi ent chaotic behavior during training\, we compute Local Lyapunov Spectra a nd observe that\, even in low-dimensional teacher-student tasks\, there is local chaoticity that can theoretically lead to exponentially diverging p arameter trajectories before they settle into stable minima or flat manifo lds.

Principal Component Analysis of these trajectories further uncovers a marked reduction in effective dimensionality over the course of training\, with the majority of variance confined to leading modes that c oincide with directions of minimal curvature in the Hessian.

Fin ally\, when extending our framework to a network learning from a dataset o n the MNIST classification task\, we find that entropy measures derived fr om positive Local Lyapunov Exponents do not correlate with generalization performance\, highlighting the need for alternative complexity metrics in realistic\, high-dimensional settings.

END:VEVENT END:VCALENDAR