11
$\begingroup$

Let $\mathbf w_1,\mathbf w_2,\ldots,\mathbf w_n \in \mathbb R^p$ and $\mathbf v \in \mathbb R^n$ be fixed vectors, and $\mathbf x \sim \mathcal N_p(\boldsymbol{\mu}, \mathbf{\Sigma})$ be an $p$-dimensional Gaussian random vector. Consider the random variable $s(\mathbf{x})$ defined by

$$sm(\mathbf{x}) := \frac{1}{1 + \sum_{i=1}^n\exp(-(\mathbf{w}_i^T\mathbf{x} + v_i))} = \frac{1}{1 -n + \sum_{i=1}^n\dfrac{1}{s(\mathbf w_i^T\mathbf x + v_i)}}, $$

where $s(a) := (1 + \exp(-a))^{-1}$ is the sigmoid function.

Question: How to (approximately) compute the expected value of $s(\mathbf x)$.

Observations: $\mathbf w_i^T \mathbf x + v_i\sim \mathcal N_1 (\mathbf w_i^T\boldsymbol{\mu} + v_i, \mathbf w_i^T \mathbf \Sigma \mathbf w_i)$, and so by virtue of this post, one has

$$\mathbb E [s(\mathbf w_i^T \mathbf x + v_i)] \approx \Phi\left(\frac{\lambda\mathbf w_i^T\boldsymbol{\mu}+ v_i}{(1 + \lambda^2 \mathbf w_i^T \mathbf \Sigma \mathbf w_i)^{1/2}}\right), $$

for some fine-tuned $\lambda > 0$ (e.g $\lambda = \pi^2 / 8$ is claimed to be "good enough"), and $\Phi(z) := \frac{1}{\sqrt{2\pi}}\int_{-\infty}^z\exp(-t^2/2)dt$ is cdf of the standard unit Gaussian $\mathcal N_1(0,1)$.

Thus one can get a (potentially very crude approximation as follows):

$$\mathbb E [sm(\mathbf x)] \approx \frac{1}{1 -n + \sum_{i=1}^n\dfrac{1}{\mathbb E [s(\mathbf w_i^T\mathbf x + v_i)]}} \approx \frac{1}{1 -n + \sum_{i=1}^n\dfrac{1}{\Phi\left(\dfrac{\lambda\mathbf w_i^T\boldsymbol{\mu}+ v_i}{(1 + \lambda^2 \mathbf w_i^T \mathbf \Sigma \mathbf w_i)^{1/2}}\right)}}. $$

$\endgroup$
4
  • 2
    $\begingroup$ It looks like by "estimate" you mean "compute, at least approximately" (as opposed to estimating it from data as a parameter in a multivariate model). According to Wikipedia, there is "no analytical solution." Numerical evaluation is relatively easy. $\endgroup$
    – whuber
    Commented Nov 24, 2017 at 15:52
  • 1
    $\begingroup$ I meant "compute, at least approximately". I've edited the question to make this clearer. $\endgroup$
    – dohmatob
    Commented Nov 24, 2017 at 16:08
  • 1
    $\begingroup$ @whuber: I just checked your wikipedia reference and it's indeed quite relevant to my question. For example the case $n=1$ of my problem indeed corresponds to a logit-normal distribution. I'll try to link the general case to the multivariate logit-normal distro and hope to recover the sought-for estimates this way $\endgroup$
    – dohmatob
    Commented Nov 24, 2017 at 16:38
  • $\begingroup$ where does this softmax definition come from? $\endgroup$ Commented Oct 31, 2018 at 10:10

1 Answer 1

-1
$\begingroup$

Late to the party, but there's another (perhaps less complicated) avenue to calculate the value of $s(x)$, and that's taking the average of $s(x)$ over a very large number of runs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.