[POSTPONED] Large-Scale Cyber-Physical Systems: From Control Theory to Deep Learning

Abstract:

The expansion of large-scale cyber-physical systems such as electrical grids, transportation networks, IoT, and other societal networks has created enormous challenges for controlling them and, at the same time, tremendous opportunities for utilizing the massive amounts of data generated by them. At the core of these data-driven control problems are distributed and stochastic optimization algorithms, such as the widely used stochastic gradient descent (SGD). While easy to use, these algorithms are not so easy to analyze, and in many cases, the reasons behind their good properties, such as their remarkable generalization performance in deep learning, have been little understood. In an attempt to shed light on the behavior of these algorithms, we revisit some minimax properties and a fundamental identity of SGD, for the square loss of linear models—developed in control theory in the 1990’s—and extend them to general loss functions and general nonlinear models. I show how this identity can be used to establish convergence of SGD to “special” global minima in highly-overparameterized nonlinear settings, a phenomenon referred to as implicit regularization. Further, I will present an approach for controlling the form of implicit regularization using the family of stochastic mirror descent (SMD) algorithms, of which SGD is a special case. The results I present include both characterization theorems and an experimental exploration of the implicit regularization of SMD algorithms. Beyond stochastic optimization algorithms, I will also highlight some of the other aspects of my work on large-scale cyber-physical systems, namely, designing markets for the smart grid, controlling epidemics in complex networks, and distributed optimization algorithms.

Bio:

Navid Azizan is a fifth-year Ph.D. candidate in Computing and Mathematical Sciences (CMS) at the California Institute of Technology (Caltech), where he is co-advised by Babak Hassibi and Adam Wierman. He was named an Amazon Fellow in Artificial Intelligence in 2017 and a PIMCO Fellow in Data Science in 2018. Additionally, he was a research intern at Google DeepMind in 2019. He also received the ITA Graduation-Day Gold Award in 2020. His research on electricity markets received the ACM GREENMETRICS Best Student Paper Award in 2016. He was also the first-place winner and a gold medalist at the 2008 National Physics Olympiad in Iran. His research interests broadly lie in mathematical optimization, control theory, machine learning, and networks. His work has focused on the design and analysis of optimization algorithms for nonconvex and networked problems, with real-world applications such as in deep learning, energy markets, distributed computation, and social networks. He received the B.Sc. degree from Sharif University of Technology and the M.Sc. degree from the University of Southern California, in 2013 and 2015, respectively.