Estimating maximum predictive power in noisy data

Question

When assessing model performance, one would like to separate prediction errors due to limitations of the model from those errors due to intrinsic noise. For example, in noisy data, AUC in an ROC curve can be less than 1.0 even when using the "true" model. What are some approaches to get a quantitative estimate of a model's maximum predictive power given a certain level of noise?

Answer 1

Bayes error rate quantifies a lower bound on classification error given the inherent overlap/noise between classes. For the two-class case let $\Omega_1, \Omega_2$ be how the model partitions the sample space and $C_i, i=1,2$ be the true class labels. Then

$\begin{align} berr &= \mathbf{P}(x\in \Omega_2,C_1) + \mathbf{P}(x\in \Omega_1,C_2) \\ &= \int_{\Omega_2} \mathbf{P}(x|C_1) \mathbf{P}(C_1)dx + \int_{\Omega_1} \mathbf{P}(x|C_2) \mathbf{P}(C_2)dx \end{align}$

So you are best off classifying $x$ as class 1 if $\frac{\mathbf{P}(x|C_1)}{\mathbf{P}(x|C_2)} > \frac{\mathbf{P}(C_2)}{\mathbf{P}(C_1)}$ which is what a bayes error rules does.

The key is identifying $\mathbf{P}(x|C_k)$, a logistic regression model will give you these values but what the best model can produce depends on the predictors. So in general, unless you know the correct model you can't know Bayes error. You can sample models from your predictors to get a distribution and rough idea.