Abstract: 
The empirical risk minimization problem with relative entropy regularization (ERM-RER) is presented considering that the reference measure is a $\sigma$-finite measure instead of a probability measure. This generalization allows for a larger degree of flexibility in the incorporation of prior knowledge over the set of models. We discuss the interplay of the regularization parameter, the reference measure, the risk measure, and the expected empirical risk induced by the solution of the ERM-RER problem, which is proved to be unique. We show that the expectation of the sensitivity is upper bounded, up to a constant factor, by the square root of the lautum information between the models and the datasets. Using these tools, dataset aggregation is studied and different figures of merit to evaluate the generalization capabilities of ERM-RER are introduced. For arbitrary datasets and parameters of the ERM-RER solution, a connection between Jeffrey’s divergence, training, and test error is established. We conclude by extending the results to $f$-divergence regularization by obtaining a closed form expression for the solution under mild assumptions on the structure of the regularizer. This analytical solution is leveraged to characterize the sensitivity of the resulting supervised learning problem and we evaluate the solution for specific regularizers arising in estimation, high-dimensional statistics, and hypothesis testing. 

Biography:
Iñaki Esnaola is a Senior Lecturer (Associate Professor) at the Department of Automatic Control and Systems Engineering at the University of Sheffield, UK, and a Visiting Research Collaborator in the Department of Electrical Engineering at Princeton University, NJ, USA. He obtained a combined BSc and MSc in Telecommunication Engineering from University of Navarra, Spain, in 2006 and the PhD in Electrical Engineering from University of Delaware, DE, USA, in 2011. In 2010-2011 he was a Research Intern at Bell Laboratories, Alcatel-Lucent, Holmdel, NJ, USA, and in 2011-2013 he was a Postdoctoral Research Associate at Princeton University. His research lies at the broad interface of information theory, communication theory, and high-dimensional statistics with an emphasis on the application to cyber-physical systems. More specifically, he studies the fundamental limits governing robust data acquisition and state estimation for systems operating with incomplete or mismatched model descriptions.