Gaussian functions model probabilities directly, yet they are used a lot less than logistic activation functions. Does anyone have any ideas why Gaussian's aren't used?
$\begingroup$
$\endgroup$
5
-
4$\begingroup$ A good deal slower to calculate, for one thing; also in the far tail the faster approaches to 0 and 1 may affect its attractiveness. $\endgroup$– Glen_bCommented Mar 10, 2016 at 3:37
-
$\begingroup$ Slower to calculate? That is, computing the derivative? $\endgroup$– user46925Commented Mar 10, 2016 at 17:20
-
3$\begingroup$ The normal cdf (at least in some implementations/on some platforms) itself will be substantially slower to evaluate that the logistic function. Its derivative won't be much different from the one for the logistic. $\endgroup$– Glen_bCommented Mar 11, 2016 at 7:10
-
1$\begingroup$ It looks like there's some confusion about what you're asking. Are you asking why Gaussian's aren't used in the hidden layers of neural networks, or why they aren't used at the output layer? The former isn't done due to Glen_b's comments. The latter is perfectly reasonable if you expect your output to be normally distributed, and isn't much more computationally expensive than a softmax. However it sounds like you're saying you want to model probabilities, in which case the Gaussian is not a good choice since its output is not between 0 and 1. $\endgroup$– Alex R.Commented Sep 7, 2017 at 18:09
-
$\begingroup$ Not answering exactly what you are asking, but maybe you want to look at the probabilistic neural networks. $\endgroup$– amanitaCommented Sep 7, 2017 at 22:21
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
@Glen_b notes in comments that it's expensive to compute exact Normal CDF probabilities. Accurate approximations exist, but one still must ask Why do this?
Logit and probit functions look basically the same, up to rescaling.
(Image reproduced from @gung's answer here.)
From a practical standpoint, it's hard see how the small differences between the two would be significant. Moreover, recent work on ReLUs (and similar) activation functions have found substantial improvement over logistic units. Due to the similarity of logistic and probit units, we can surmise that ReLUs will likewise also out-perform probit units.
All together, then, probit units are a more expensive way to get worse results than ReLU (or its variants).
answered Jun 30, 2018 at 13:52