I have about 10 classes, on which I train a CNN with a softmax output layer using one-hot encoding and categorical cross-entropy loss.
The problem is that two pairs of these of these classes (let's say A and B, and also E and F) also have very few "in between" examples. So I would expect the following possible scenarios:
- Input clearly belongs to a class from A to J
- Input belongs to A,B or input belongs to E,F
I see three options here.
- Introduce distinct output classes A&B, E&F. The issue is that I have relatively few examples here (possibly fixable), but I'm mainly worried about possible confusion - would the classifier be able to differentiate between A, B, A&B?
- Add training examples where the expected feature vector is [0.5, 0.5, 0, ...]. Theoretically this would make the ouptut close to 0.5 for the classes of interest, which I'm happy with.
- Change the output from softmax to sigmoid and treat it as a fully multilabel classification. I'm a bit worried that this would impact the overall accuracy, considering that I expect less than 5% of the inputs to be "in between".
Theoretically, I'd say all my options would work, but I can't tell their pros and cons.