4
$\begingroup$

I am building a Neural Network for a binary classification problem where the Bayes error (lowest possible error rate) is probably close to 50%.

What makes the task easier is that I don't need to make a prediction for each observation of the test sample. I only want to make a prediction for the observations where the model has a fairly high confidence. However a high rate at which predictions are made is better than a low one.

So far, I have used a standard neural network (feed-forward, cross-entropy loss, L2 regularization and sigmoid activation on final node). In the testing sample, I only take into account the observations for which the final node's value $(\hat{Y}_i)$ is outside of an interval of low confidence: $$\text{predicted class}_i = \begin{cases} 1 &\text{ if } \hat{Y}_i > 0.5 + a \\ 0 &\text{ if } \hat{Y}_i < 0.5 - a \\ \text{NA} &\text{else} \end{cases} \\ \text{where } a\in [0, 0.5] \text{ indicates the level of confidence required}$$

To tune the hyperparameters (including $a$), I have designed a metric that depends positively on:

  • Test-sample accuracy (only counting predictions different from NA)
  • Percentage of predictions that are different from NA.

I am not yet satisfied with the performance achieved with this approach, and I am sure that there are smarter ways to approach this, for example a custom loss function. Advices, links to articles, or even related search keywords are welcome.

$\endgroup$
0
$\begingroup$

You might find focal loss interesting. This is a reshaped standard cross entropy loss that down-weights the loss assigned to well-classified examples. It motivates a classifier to show more confidence where appropriate instead of only fearing a huge penalty for misclassification and hiding behind the base rate.

It is also possible that the high Bayes error prevents good learning of useful features. Coming up with a toy task as an intermediate step might help.

Curious to hear if you made further progress.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.