I am currently reading the paper Fast learning rates for plug-in classifiers under the margin condition by Audibert and Tsybakov (2014), in which the authors prove that, for binary classification problems, plug-in classifiers can achieve fast convergence rates under some assumptions on the data distribution.

More precisely, for a binary classification problem with feature space $\mathcal X = \mathbb R^d$ and output space $\mathcal Y = \{0,1\}$, and the Bayes regression function defined as $\eta(x) := \mathbb P(Y=1|X=x) $, the authors define the plug-in classifiers as $$\hat f^{PI}(X) = \mathbf 1_{\{\hat \eta_n(X)\ge\frac 1 2\}} $$ Where $\hat \eta_n$ is an estimator of the regression function $\eta$.

A lot of the discussion throughout the paper consists in comparing the convergence rate of the excess risk of these plug-in classifiers with the excess risk of Empirical Risk Minimization classifiers, which according to Wikipedia have the following expression : $$\hat f^{ERM} = \underset{f\in\mathcal F_n}{\mathrm{argmin}} \frac 1 n \sum_{i=1}^n L(f(X_i),y_i)$$ Where $L$ is a loss function and $\mathcal F_n$ is a class of functions for the prediction rule.

I am however quite confused because it seems to me that plug-in classifiers are in fact also Empirical Risk Minimizers : Indeed, in practice, the regression function is learned by minimizing some loss depending on the datapoints, so the process of learning the regression function and thus the plug-in classifier can be thought of as a special case of Empirical Risk Minimization.

Can someone explain to me the difference between the two ?


After reading more carefully, turns out the answer was written in plain text : The estimator $\hat \eta_n$ of the regression function $\eta$ is non-parametric, it could for instance be a Kernel Density Estimator, a k-NN Estimator, a local polynomial estimator, or many other things...
Point is, $\hat \eta_n$ is not "learned" by minimizing a loss objective, and it turns out that the two frameworks (Empirical Risk Minimization vs Plug-In) are actually quite different.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.