2
$\begingroup$

When assessing model performance, one would like to separate prediction errors due to limitations of the model from those errors due to intrinsic noise. For example, in noisy data, AUC in an ROC curve can be less than 1.0 even when using the "true" model. What are some approaches to get a quantitative estimate of a model's maximum predictive power given a certain level of noise?

$\endgroup$
4
$\begingroup$

Bayes error rate quantifies a lower bound on classification error given the inherent overlap/noise between classes. For the two-class case let $\Omega_1, \Omega_2$ be how the model partitions the sample space and $C_i, i=1,2$ be the true class labels. Then

$\begin{align} berr &= \mathbf{P}(x\in \Omega_2,C_1) + \mathbf{P}(x\in \Omega_1,C_2) \\ &= \int_{\Omega_2} \mathbf{P}(x|C_1) \mathbf{P}(C_1)dx + \int_{\Omega_1} \mathbf{P}(x|C_2) \mathbf{P}(C_2)dx \end{align}$

So you are best off classifying $x$ as class 1 if $\frac{\mathbf{P}(x|C_1)}{\mathbf{P}(x|C_2)} > \frac{\mathbf{P}(C_2)}{\mathbf{P}(C_1)}$ which is what a bayes error rules does.

The key is identifying $\mathbf{P}(x|C_k)$, a logistic regression model will give you these values but what the best model can produce depends on the predictors. So in general, unless you know the correct model you can't know Bayes error. You can sample models from your predictors to get a distribution and rough idea.

$\endgroup$
  • $\begingroup$ Bayes error is the right theoretical view, but as noted we cannot know this in real data. There are approaches that try to estimate the noise ceiling from real data, e.g., Quantifying variability in neural responses and its application for the validation of model predictions. Network. 2004. Hsu A, Borst A, Theunissen FE.[link] (ahsu.psychol.ucl.ac.uk/ahsu/papers/quantifying_variability.PDF) $\endgroup$ – bythemark Aug 21 '12 at 23:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.