Timeline for Tensorflow Cross Entropy for Regression?
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 16, 2021 at 5:50 | comment | added | Allohvk | This need not be restricted to output range of [0,1] only. Users of deep models prefer cross entropy over MSE. I have seen non [0,1] regression output being compressed to [0,1] using a sigmoid just to use cross entropy loss function after that. Of course later on we need to apply a logit function later on the predicted values to get back the original range. Dropout, BatchNorm etc are more stable with Cross Entropy loss and in general the convergence is also faster | |
| Nov 21, 2018 at 15:14 | comment | added | today | @InfProbSciX Thanks for your reply. So as you mentioned, depending on the regression task (and the assumptions on the distribution of data, errors, etc.) a loss function might not be reasonable to be used. And, as I mentioned, this is true for all loss functions, including crossentropy. Of course, I see your point that just because the output values are in the range [0,1] does not guarantee that crossentropy is the optimal choice loss function and I was not trying to convey the otherwise in my answer. | |
| Nov 21, 2018 at 14:46 | comment | added | adityar | The way I'd define reasonable is by constructing a model law. For example, in a regression framework such as $Y = f_{\theta}(X) + \epsilon$ where $\epsilon$ are i.i.d. errors - say normally distributed, the negative log-likelihood is exactly the squared loss. In a setting where the model law looks like $Y \sim Bernoulli(p_{\theta})$, the negative log-likelihood is exactly the binary cross entropy. Where the law is a linear regression with a normal prior on the coefs, the loss corresponds to the L2 penalty and so on. Where possible, I'd construct a law and then derive a loss. | |
| Nov 21, 2018 at 14:41 | comment | added | today | @InfProbSciX "it might not be a reasonable approach to deal with any regression where the outputs are in a [0,1] range." So "reasonable" in what sense? Or how do you define the reasonability of loss function for a specific task? I suspect that statement might be true for any loss function. Is there any loss function that would be reasonable to use for all kinds of regression tasks, of course after defining the "reasonable"? | |
| Nov 21, 2018 at 14:36 | comment | added | adityar | Loss functions can viewed as likelihoods / posteriors or some monotonic transformation of them. So, while it is true that in some regression models a loss similar to the cross-entropy might make sense, it might not be a reasonable approach to deal with any regression where the outputs are in a $[0, 1]$ range. | |
| Nov 21, 2018 at 14:06 | history | answered | today | CC BY-SA 4.0 |