Abstract: Privacy enhancing technologies, such as differentially private stochastic gradient descent (DP-SGD), allow us to access private data without worrying about leaking sensitive information. This is crucial in the modern era of data-centric AI, where all public data has been exhausted and the next frontier models rely on access to high-quality data. A central component in these technologies is private statistical estimation, such as mean estimation and linear regression. We present a series of results where robust statistics and stable…

Abstract: Training data determines what neural networks can learn—but can we predict the geometry of learned representations directly from data statistics? We  present a framework that addresses this question for sufficiently large, well-trained neural networks. The key idea is a coarse but predictive abstraction of such networks as log-bilinear softmax models, whose implicit regularization we can analyze. Within this framework, we show how label imbalance shapes representation geometry and, for language models, how word and context representations organize into structures…