Let $\mathbf w_1,\mathbf w_2,\ldots,\mathbf w_n \in \mathbb R^p$ and $\mathbf v \in \mathbb R^n$ be fixed vectors, and $\mathbf x \sim \mathcal N_p(\boldsymbol{\mu}, \mathbf{\Sigma})$ be an $p$-dimensional Gaussian random vector. Consider the random variable $s(\mathbf{x})$ defined by

$$sm(\mathbf{x}) := \frac{1}{1 + \sum_{i=1}^n\exp(-(\mathbf{w}_i^T\mathbf{x} + v_i))} = \frac{1}{1 -n + \sum_{i=1}^n\dfrac{1}{s(\mathbf w_i^T\mathbf x + v_i)}}, $$

where $s(a) := (1 + \exp(-a))^{-1}$ is the sigmoid function.

Question: How to (approximately) compute the expected value of $s(\mathbf x)$.

Observations: $\mathbf w_i^T \mathbf x + v_i\sim \mathcal N_1 (\mathbf w_i^T\boldsymbol{\mu} + v_i, \mathbf w_i^T \mathbf \Sigma \mathbf w_i)$, and so by virtue of this post, one has

$$\mathbb E [s(\mathbf w_i^T \mathbf x + v_i)] \approx \Phi\left(\frac{\lambda\mathbf w_i^T\boldsymbol{\mu}+ v_i}{(1 + \lambda^2 \mathbf w_i^T \mathbf \Sigma \mathbf w_i)^{1/2}}\right), $$

for some fine-tuned $\lambda > 0$ (e.g $\lambda = \pi^2 / 8$ is claimed to be "good enough"), and $\Phi(z) := \frac{1}{\sqrt{2\pi}}\int_{-\infty}^z\exp(-t^2/2)dt$ is cdf of the standard unit Gaussian $\mathcal N_1(0,1)$.

Thus one can get a (potentially very crude approximation as follows):

$$\mathbb E [sm(\mathbf x)] \approx \frac{1}{1 -n + \sum_{i=1}^n\dfrac{1}{\mathbb E [s(\mathbf w_i^T\mathbf x + v_i)]}} \approx \frac{1}{1 -n + \sum_{i=1}^n\dfrac{1}{\Phi\left(\dfrac{\lambda\mathbf w_i^T\boldsymbol{\mu}+ v_i}{(1 + \lambda^2 \mathbf w_i^T \mathbf \Sigma \mathbf w_i)^{1/2}}\right)}}. $$

  • 2
    $\begingroup$ It looks like by "estimate" you mean "compute, at least approximately" (as opposed to estimating it from data as a parameter in a multivariate model). According to Wikipedia, there is "no analytical solution." Numerical evaluation is relatively easy. $\endgroup$ – whuber Nov 24 '17 at 15:52
  • 1
    $\begingroup$ I meant "compute, at least approximately". I've edited the question to make this clearer. $\endgroup$ – dohmatob Nov 24 '17 at 16:08
  • $\begingroup$ @whuber: I just checked your wikipedia reference and it's indeed quite relevant to my question. For example the case $n=1$ of my problem indeed corresponds to a logit-normal distribution. I'll try to link the general case to the multivariate logit-normal distro and hope to recover the sought-for estimates this way $\endgroup$ – dohmatob Nov 24 '17 at 16:38
  • $\begingroup$ where does this softmax definition come from? $\endgroup$ – Mr Tsjolder Oct 31 '18 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.