11
$\begingroup$

Both the logistic function and standard deviation are usually denoted $\sigma$. I'll use $\sigma(x) = 1/(1+\exp(-x))$ and $s$ for standard deviation.

I have a logistic neuron with a random input whose mean $\mu$ and standard deviation $s$ I know. I hope the difference from the mean can be approximated well by some Gaussian noise. So, with a slight abuse of notation, assume it produces $\sigma(\mu + N(0,s^2))=\sigma(N(\mu,s^2))$. What is the expected value of $\sigma(N(\mu,s^2))$? The standard deviation $s$ might be large or small compared with $\mu$ or $1$. A good closed form approximation for the expected value would be almost as good as a closed form solution.

I don't think a closed form solution exists. This can be viewed as a convolution, and the characteristic function for the logistic density is known ($\pi t ~\text{csch} ~\pi t$), but I'm not sure how much that helps. The inverse symbolic calculator was unable to recognize the density at $0$ of the convolution of the logistic distribution's density and a standard normal distribution, which suggests but does not prove that there is no simple elementary integral. More circumstantial evidence: In some papers on adding Gaussian input noise to neural networks with logistic neurons, the papers didn't give closed form expressions either.

This question arose in trying to understand the error in the mean field approximation in Boltzman machines.

$\endgroup$
0

2 Answers 2

6
$\begingroup$

The following is what I ended up using:

Write $\sigma(N(\mu,s^2)) = \sigma(\mu + X)$ where $X \sim N(0,s^2)$. We can use a Taylor series expansion.

$\sigma(\mu + X) = \sigma(\mu) + X \sigma'(\mu) + \frac{X^2}{2} \sigma''(\mu)+ ... + \frac{X^n}{n!}\sigma^{(n)}(\mu) + ...$

$\begin{eqnarray} E[\sigma(\mu + X)] & =& E[\sigma(\mu)] + E[X \sigma'(\mu)] + E[\frac{X^2}{2} \sigma''(\mu)] + ... \newline & = & \sigma(\mu) + 0 + \frac{s^2}{2}\sigma''(\mu) + 0 + \frac{3s^4}{24}\sigma^{(4)}(\mu)+ ... + \frac{s^{2k}}{2^k k!}\sigma^{(2k)}(\mu) ... \end{eqnarray}$

There are convergence issues. The logistic function has a pole where $\exp(-x) = -1$, so at $x = k \pi i$, $k$ odd. Divergence is not the same thing as the prefix being useless, but this series approximation may be unreliable when $P(|X| \gt \sqrt{\mu^2 + \pi^2})$ is significant.

Since $\sigma'(x) = \sigma(x) (1-\sigma(x))$, we can write derivatives of $\sigma(x)$ as polynomials in $\sigma(x)$. For example, $\sigma'' = \sigma-3\sigma^2+2\sigma^3$ and $\sigma''' = \sigma - 7\sigma^2 + 12 \sigma^3 - 6\sigma^4$. The coefficients are related to OEIS A028246.

$\endgroup$
5
$\begingroup$

What you have here is a random variable that follows a logit-normal (or logistic-normal) distribution (see wikipedia), that is, $\mbox{logit}[x] \sim N(\mu, s^2)$. The moments of the logit-normal distribution do not have analytical solutions.

But of course one can get them via numerical integration. If you use R, there is the logitnorm package that has everything you need. An example:

install.packages("logitnorm")
library(logitnorm)
momentsLogitnorm(mu=1, sigma=2)

This yields:

> momentsLogitnorm(mu=1, sigma=2)
      mean        var 
0.64772644 0.08767866

So, there is even a convenience function that will directly give you the mean and variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.