Neural Networks - Strategies for problems with high Bayes error rate

Question

I am building a Neural Network for a binary classification problem where the Bayes error (lowest possible error rate) is probably close to 50%.

What makes the task easier is that I don't need to make a prediction for each observation of the test sample. I only want to make a prediction for the observations where the model has a fairly high confidence. However a high rate at which predictions are made is better than a low one.

So far, I have used a standard neural network (feed-forward, cross-entropy loss, L2 regularization and sigmoid activation on final node). In the testing sample, I only take into account the observations for which the final node's value $(\hat{Y}_i)$ is outside of an interval of low confidence: $$\text{predicted class}_i = \begin{cases} 1 &\text{ if } \hat{Y}_i > 0.5 + a \\ 0 &\text{ if } \hat{Y}_i < 0.5 - a \\ \text{NA} &\text{else} \end{cases} \\ \text{where } a\in [0, 0.5] \text{ indicates the level of confidence required}$$

To tune the hyperparameters (including $a$), I have designed a metric that depends positively on:

• Test-sample accuracy (only counting predictions different from NA)
• Percentage of predictions that are different from NA.

I am not yet satisfied with the performance achieved with this approach, and I am sure that there are smarter ways to approach this, for example a custom loss function. Advices, links to articles, or even related search keywords are welcome.

Answer 1

You might find focal loss interesting. This is a reshaped standard cross entropy loss that down-weights the loss assigned to well-classified examples. It motivates a classifier to show more confidence where appropriate instead of only fearing a huge penalty for misclassification and hiding behind the base rate.

It is also possible that the high Bayes error prevents good learning of useful features. Coming up with a toy task as an intermediate step might help.

Curious to hear if you made further progress.