Binary Classification with a third 'uncertain' class label

Question

Consider the task of classifying an image into two classes:

1. Image shows a cat;
2. Image shows no cat.

A data set is provided for training/testing a binary classifier. However, three labels are provided for each image in the data set:

1. Image shows a cat;
2. Image shows no cat;
3. Undecided.

The third class label (undecided) implies that the image is of bad quality, i.e., it is impossible to determine with confidence that the image shows either (1) a cat or (2) no cat. An example is a very blurry image.

My initial approach to solving the task was to discard the third (undecided) class label and train/test a binary classifier with the first (cat) and second (no cat) class labels, since the original task requires classifying an image as either (1) showing a cat or (2) showing no cat . However, this will reduce the size of the data set significantly. I now have the following questions:

Question 1: Consider merging the third class label (undecided) into the second class label (no cat) such that the labels in the data set would be split as follows:

1. Image shows a cat;
2. Image shows no cat or undecided.

What are the implications of training a binary classifier with this data set?

Question 2: Consider changing the original task into multiclass classification, where an image is classified into three classes:

1. Image shows a cat;
2. Image shows no cat;
3. Undecided.

What are the implications of training a multiclass classifier with this data set? Can an image be classified as 'undecided'?

Answer 1

Question 2 is easier to answer. Your model will learn that undecided is something different than cat (or no cat). Even when a picture clearly shows a cat, if the image even more clearly is blurry (or whatever else leads to "undecided"), the model may give a higher probability to "undecided" than to "cat". I'm not sure whether that's what you want, you may e.g. be interested to have a model that can classify lower quality images into cat/no cat, even humans cannot do it (or struggle), which you would (likely) not get with this option.

Question 1 makes sense if you are pretty sure that the vast majority of undecided images is "no cat"-images. You could also take a label-smoothing approach, e.g. by instead of 1 (cat) or 0 (no cat), you could use 0.1 (~10% chance this could be a cat) or 0.2 for the undecided ones depending on what you believe of them (you could even have human assigned probabilities for each image).

You could also make this two separate binary labels: 1) cat vs. no cat and 2) decided (either classified as cat or not) vs. "not decided". If you do that, you could specify that no loss should be incurred for cat vs. no cat for images from the undecided category. However, I guess this might not gain you much for cat vs. no cat.

Or you could do self-distillation, i.e. start with one of the various possible approaches (including the one where you initially don't use the "undecided" images), then train your model, then soft-label (assign model predicted probability between 0 and 1 to the undecided images) the undecided images and train again now including the soft-labelled images. Generally, I'm not sure how much this would really help you for classifying cat vs. no cat, but I'd guess that this will only help you if you have a large number of undecided images (i.e. substantially more than "decided" images). Similarly, just a large amount of unlabeled images can be used the same way and helps a lot (see e.g. here https://arxiv.org/abs/1911.04252).