When assessing model performance, one would like to separate prediction errors due to limitations of the model from those errors due to intrinsic noise. For example, in noisy data, AUC in an ROC curve can be less than 1.0 even when using the "true" model. What are some approaches to get a quantitative estimate of a model's maximum predictive power given a certain level of noise?


1 Answer 1


Bayes error rate quantifies a lower bound on classification error given the inherent overlap/noise between classes. For the two-class case let $\Omega_1, \Omega_2$ be how the model partitions the sample space and $C_i, i=1,2$ be the true class labels. Then

$\begin{align} berr &= \mathbf{P}(x\in \Omega_2,C_1) + \mathbf{P}(x\in \Omega_1,C_2) \\ &= \int_{\Omega_2} \mathbf{P}(x|C_1) \mathbf{P}(C_1)dx + \int_{\Omega_1} \mathbf{P}(x|C_2) \mathbf{P}(C_2)dx \end{align}$

So you are best off classifying $x$ as class 1 if $\frac{\mathbf{P}(x|C_1)}{\mathbf{P}(x|C_2)} > \frac{\mathbf{P}(C_2)}{\mathbf{P}(C_1)}$ which is what a bayes error rules does.

The key is identifying $\mathbf{P}(x|C_k)$, a logistic regression model will give you these values but what the best model can produce depends on the predictors. So in general, unless you know the correct model you can't know Bayes error. You can sample models from your predictors to get a distribution and rough idea.

  • $\begingroup$ Bayes error is the right theoretical view, but as noted we cannot know this in real data. There are approaches that try to estimate the noise ceiling from real data, e.g., Quantifying variability in neural responses and its application for the validation of model predictions. Network. 2004. Hsu A, Borst A, Theunissen FE.[link] (ahsu.psychol.ucl.ac.uk/ahsu/papers/quantifying_variability.PDF) $\endgroup$
    – bythemark
    Aug 21, 2012 at 23:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.