1
$\begingroup$

I have a CNN built with Keras, based on a multilayer perceptron. The last layer is softmax.

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 248, 248, 32)      896       
_________________________________________________________________
activation_1 (Activation)    (None, 248, 248, 32)      0         
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 124, 124, 32)      0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 122, 122, 32)      9248      
_________________________________________________________________
activation_2 (Activation)    (None, 122, 122, 32)      0         
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 61, 61, 32)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 59, 59, 64)        18496     
_________________________________________________________________
activation_3 (Activation)    (None, 59, 59, 64)        0         
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 29, 29, 64)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 53824)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 64)                3444800   
_________________________________________________________________
activation_4 (Activation)    (None, 64)                0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 64)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 9)                 585       
_________________________________________________________________
activation_5 (Activation)    (None, 9)                 0         
=================================================================
Total params: 3,474,025
Trainable params: 3,474,025
Non-trainable params: 0
_________________________________________________________________

Now, during classification, there are some cases in which I know in advance my input is not class A, or B, or C (because of some specific business logic).

Example : I have an image X that I want to classify. My CNN can classify between 10 different classes. I know, without doing anything, that my input is not class A or class B.

Is there any way I could theoretically take advantage of this knowledge to help the classification process ?

Since the last layer is in softmax, I was thinking of setting the values of the output neurons I know are not the right answer to 0 and redistribute this value among potentially right neurons.

I hope my question was clear enough.

$\endgroup$
  • $\begingroup$ You could train a separate model that has only learned to distinguish between those two classes. Since it has a more specific goal, given enough examples it might outperform the three-classes model. $\endgroup$ – Frans Rodenburg Apr 26 at 11:46
  • $\begingroup$ The thing is, class A or B were examples, but this could be different. At each classification, I might know in advance one or more classes that my input cannot be an instance of. But that's variable, and it could be different classes $\endgroup$ – Arthur Attout Apr 26 at 12:26
1
$\begingroup$

Typically you have $P(y|x) = \text{softmax} (f(x;\theta))$, where $f$ is your network. But given a mask $m$ of invalid classes, you can simply say $P(y|x,m) = \text{softmax}(f(x;\theta) - km)$, where $k$ is set to some very large constant. And this works fine both at train time and at test time.

Since the last layer is in softmax, I was thinking of setting the values of the output neurons I know are not the right answer to 0 and redistribute this value among potentially right neurons.

The approach described above is basically this, except there is no need to redistribute the pre-softmax scores at all. The post-softmax probabilities would be automatically "redistributed" by virtue of softmax's normalizing properties.

$\endgroup$
  • 1
    $\begingroup$ Okay, that's pretty close to what I was expecting. Unfortunately this method wil basically give weight to neurons that would potentially be "just noise". I thought there would be a way to take full advantage of the current neural network training, and give some more knowledge to help classification, but I think there's no way around this. Thanks ! $\endgroup$ – Arthur Attout May 8 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.