Sigmoid activation is used with the binary_cross_entropy loss where the output is a binary classification (yes/no). I often have outputs which are bounded between [0-1], but aren't classification outputs. Eg, maybe it's a real number which I've normalized to that range; or maybe it's a confidence score. I always see recommended to use linear activation for this (and mse/rmse/mae loss). But a benefit I see in using sigmoid is it bounds the output, so that the DNN isn't trying all sorts of values outside the acceptable range. Seems like it would train faster this way, and ensure the outputs are [0-1] on inference. So my intuition is: faster, safer.

But with the sigmoid activation not being linear in nature, will that cause strangeness in the way the output distribution is fitted? Ie, is sigmoid indeed a bad idea re: the output distribution, and I should stick with linear?



Browse other questions tagged or ask your own question.