Date:
Thu, 29/06/202313:00-14:00
The seminar will take place in Ross 70.
Title: High Dimensional Limit of Stochastic Gradient Decent of Non-Convex Problems
Abstract: Stochastic Gradient Descent (SGD) has become a widely employed optimization technique in modern machine learning algorithms, enabling the optimization of millions or even billions of parameters. Analyzing SGD mathematically in the high-dimensional regime poses significant challenges, particularly when dealing with nonsmooth and non-convex objective functions. In this talk, I will present a high-dimensional limit of SGD denoted as homogenized SGD (HSGD), of several nonlinear and nonconvex machine learning models such as generalized linear model, phase retrieval, and multi-class logistic regression. I will show that HSGD converges to SGD for a large class of statistic functions when the number of samples n is commensurate with the number of features d. Leveraging this limit, precise non-asymptotic expressions for the generalization performance of SGD in high-dimensional settings can be derived. These expressions are formulated in terms of a solution to a Volterra integral equation, offering exact insights into the generalization capabilities of SGD. Finally, I will show that HSGD can also be used to provide convergence guarantees of SGD in certain scenarios. This is joint work with Elizabeth Collins-Woodfin, Courtney Paquette, and Elliot Paquette.
Title: High Dimensional Limit of Stochastic Gradient Decent of Non-Convex Problems
Abstract: Stochastic Gradient Descent (SGD) has become a widely employed optimization technique in modern machine learning algorithms, enabling the optimization of millions or even billions of parameters. Analyzing SGD mathematically in the high-dimensional regime poses significant challenges, particularly when dealing with nonsmooth and non-convex objective functions. In this talk, I will present a high-dimensional limit of SGD denoted as homogenized SGD (HSGD), of several nonlinear and nonconvex machine learning models such as generalized linear model, phase retrieval, and multi-class logistic regression. I will show that HSGD converges to SGD for a large class of statistic functions when the number of samples n is commensurate with the number of features d. Leveraging this limit, precise non-asymptotic expressions for the generalization performance of SGD in high-dimensional settings can be derived. These expressions are formulated in terms of a solution to a Volterra integral equation, offering exact insights into the generalization capabilities of SGD. Finally, I will show that HSGD can also be used to provide convergence guarantees of SGD in certain scenarios. This is joint work with Elizabeth Collins-Woodfin, Courtney Paquette, and Elliot Paquette.