0
$\begingroup$

I have trained a neural network on DNA sequences data and my training set has exactly the same number of data in both classes. When I select a softmax function at the end, my accuracy remains at 47% and loss for both validation and training stays the same at around 7.6 regardless of how many batches and epochs I choose. But once I change the softmax function to sigmoid, the validation accuracy starts at 50% for the first epoch and reaches above 98% at the end which is odd cause I think at best my network should achieve an accuracy of around 80% since I know some of my data is misclassified. Why is this happening?

$\endgroup$
4
  • $\begingroup$ What exactly are you trying to predict? Softmax vs sigmoid are completely different models. $\endgroup$
    – Tim
    Apr 19, 2021 at 19:55
  • 1
    $\begingroup$ Do you have 2 output neurons in both cases? Are your class labels mutually exclusive? $\endgroup$
    – Sycorax
    Apr 19, 2021 at 20:15
  • $\begingroup$ @Tim I am trying to predict whether they belong to one class or the other. $\endgroup$
    – A4747
    Apr 21, 2021 at 20:04
  • $\begingroup$ @Sycorax I have two output neurons for softmax but one for sigmoid. $\endgroup$
    – A4747
    Apr 21, 2021 at 20:04

2 Answers 2

2
$\begingroup$

Using sigmoid with dummy encoded output (one binary column) vs using softmax with two one-hot encoded columns (one of the columns is equal to one, the other is zero) is mathematically equivalent and should give same results. Your likely have a bug in the code.

$\endgroup$
3
  • $\begingroup$ Do the probabilities in your two softmax nodes not have to add to $1$? $\endgroup$
    – Dave
    Apr 21, 2021 at 20:50
  • 1
    $\begingroup$ Are you sure about this? I thought it was the opposite way. Sigmoid was for when the outcomes are not mutually exclusive and softmax was for when the outcomes are mutually exclusive but both of them can be used for binary classification. $\endgroup$
    – A4747
    Apr 21, 2021 at 21:04
  • $\begingroup$ @A4747 no, sorry what I wrote was incorrect. I misunderstood was you were saying and the answer itself was misworded. Another example why you should not trust what you read over the Internet. You are correct, those two implementations should be the same, something is wrong. $\endgroup$
    – Tim
    Apr 22, 2021 at 4:51
-1
$\begingroup$

Softmax should work better for classification than sigmoid (with 2 output features in both cases). I'm guessing you have a bug in your code somewhere.

If you're getting better accuracy than you believe is reasonably possible, then you are probably overfitting and either have a model which is too complicated or (perhaps more likely) have too little data.

$\endgroup$
6
  • 2
    $\begingroup$ Hi, your conclusion (bug in code) is reasonable but your premise (softmax should work better) is incorrect. For binary classification, 2-class softmax is equivalent to sigmoid (because the softmax is constrained to a simplex). If the sigmoid output is $p$, then the probability for the other class is necessarily $1-p$, which you'd also get out of softmax. I'm not the downvote, but I suspect this is why. $\endgroup$ Apr 19, 2021 at 20:23
  • $\begingroup$ I know the model is not overfitting since the loss is decreasing as the epochs increase. $\endgroup$
    – A4747
    Apr 19, 2021 at 20:36
  • $\begingroup$ @AryaMcCarthy Yes, I see your point. I was thinking that they were using 2 outputs and not one. In that case, softmax would add the constraint that they need to add to one as opposed to the more relaxed constraint that they both need to be between 0 and 1 imposed by sigmoid. Softmax with 2 outputs should be equivalent to sigmoid with 1 output. Softmax with 1 output would always output 1 which could lead to a 50% accuracy bug. (not 47 if the classes are exactly balanced though...) $\endgroup$
    – JacKeown
    Apr 20, 2021 at 21:42
  • $\begingroup$ @A4747 Do you mean that your testing loss is decreasing or simply your training loss? $\endgroup$
    – JacKeown
    Apr 20, 2021 at 21:44
  • $\begingroup$ @JacKeown both were decreasing with sigmoid. I have noticed that when I increase the number of batches after a certain point the accuracy and loss remain stable and do not change. What is really strange for me is when I use "validation_split=0.3" for training with sigmoid I get above 98% accuracy but if I pass in "validation_data=(x_val, y_val)" to model.fit the accuracy will get as high as 65%. I have checked my code and I honestly don't think there are any problem with labeling the validation set. I have also shuffled both my training and validation dataset. $\endgroup$
    – A4747
    Apr 21, 2021 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.