1
$\begingroup$

Suppose that we are going to build a model for identifying cats and dogs with neural networks. In the common way, we have 2 neurons in the last layer.

If instead of two neurons, we put 3 neurons (but do not input any sample regarding the corresponding class), can we expect that non-dog, non-cat inputs (like flower, plane or ...) can be identified, thanks to that additional neuron?

$\endgroup$
3
  • 1
    $\begingroup$ There’s nothing wrong with “cat, dog, neither cat nor dog” as a classification — the categories are mutually exclusive. But do you have labels and examples for all 3 categories? $\endgroup$
    – Sycorax
    Feb 20, 2022 at 20:25
  • $\begingroup$ I asked it just for my knowledge. but how can we identify the third type (neither cat nor dog)? Because we cannot train the model over all kinds of objects. $\endgroup$ Feb 20, 2022 at 21:16
  • 1
    $\begingroup$ You probably can't train a model over every type of object, but anything that's not a cat or a dog is a valid example of the "neither" class. If you don't provide any examples of the class, then the model won't learn anything about it. $\endgroup$
    – Sycorax
    Feb 21, 2022 at 2:02

1 Answer 1

2
$\begingroup$

The additional output nodes will not help, if there are no training examples to go with it.

When you train a neural network with a softmax activation function in the final layer, the output nodes can be interpreted as a probability distribution over the classes. Training a neural network by any form of gradient descent with a cross-entropy loss will attempt to maximize the probability of the observed class. In other words, the ideal output is all of the probability mass on 'cat' if the training image is a cat, and all of the probability mass on 'dog' if it's a dog.

There's nothing in here to encourage putting any probability mass on the 'other' class, because you don't provide training examples of it. The network will learn to push the probability of that class down toward 0, regardless of whether the provided image is a dog or cat. This means it won't predict 'other' at test time.

Sure, I made some assumptions in there. But the general statement holds, even for other loss functions or training regimens.

$\endgroup$
1
  • $\begingroup$ Yes, I actually got the same results as you said (there is no prediction for that additional class despite I feed otherwise samples). I was just curious about that. Thanks for your help... $\endgroup$ Feb 21, 2022 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.