Consider a non-negative random variable $X\sim p(\theta)$, that is, following distribution $p$ parametrized by $\theta$.

Suppose we find a value of the parameters $\theta^*$ such that $$\mathbb E_{X\sim p(\theta)}[X]$$ is minimized.

  • Is $\mathbb E_{X\sim p(\theta)} [X^2]$ minimized for these parameters as well?

  • Otherwise, what might be sufficient and/or necessary conditions?

  • Does it make a difference whether $X$ is a function of $\theta$?

    • Edit: As kindly pointed out by whuber in the comments, $X$ is already a function of $\theta$, so assuming arbitrary variables/functions this last question is uninteresting.
  • $\begingroup$ Why would you want to minimize the mean? $\endgroup$ – Taylor Feb 12 at 12:56
  • $\begingroup$ I don't. I want to know which $\theta$ minimizes it. $\endgroup$ – broncoAbierto Feb 12 at 12:58
  • 2
    $\begingroup$ Your question is so nonspecific that the solution is unlikely to be enlightening: if you consider the functions $f_1:\theta\to E_\theta[X]$ and $f_2:\theta\to E_\theta[X^2],$ and let $\Omega(f_i)$ be the set at which $f_i$ attains a minimum, it is necessary and sufficient that $\Omega(f_1)\cap \Omega(f_2)$ be nonempty. There is no simplification possible. This abstract characterization makes it easy to construct counterexamples to your first bullet. The third bullet makes no sense, because $X$ does explicitly depend on $\theta$ and if it did not, there wouldn't be any question to ask. $\endgroup$ – whuber Feb 12 at 14:10
  • $\begingroup$ @whuber With regards to the third bullet, now I realize that for arbitrary $X$ there is no difference. That is a valid answer. Your first comment is essentially a rephrasing of my question, and the answer you give is wrong. $\endgroup$ – broncoAbierto Feb 12 at 14:42
  • 1
    $\begingroup$ @whuber $X$ need not be a function of $\theta$. The distribution of $X$ is a function of $\theta$. For example, consider the measurable space $(\mathbb R, \mathcal B)$, let $X$ be the identity map, and $P_\theta$ a distribution indexed by $\theta$. I can keep $X$ fixed while still varying the distribution of $X$ through $\theta$. $\endgroup$ – guy Feb 12 at 15:21

This is a partial answer actually. For the first subquestion, the optimizers are different, and an example should suffice I guess:

Let $X$ be a RV such that it is $\sim\text{Bern(1/3)}$ with probability $\theta$, or equal to $1/2$ with probability $1-\theta$. $E[X]=\theta/3+(1-\theta)/2$, which is minimized when $\theta=1$. However, $E[X^2]=\theta/3+(1-\theta)/4$ is minimized when $\theta=0$. So, the optimizers for $E[X]$ and $E[X^2]$ yield different $\theta^*$.

Here, I don't have an answer for the sufficient/necessary conditions where they're equal; and I'm not sure if it can be found or not.

For your last question, if $X$ is a function of $\theta$, then $\theta$ should be a random variable. But, we treat it as if it is an unknown constant and can be set to any value (based on its domain) depending on your objective function (e.g. minimizing $E[X]$). I can't make sense of the situation where $X=f(\theta)$, and we set $\theta$ to optimize $E[X]$.

Note: I think your notation should be $\mathbb E_{X\sim p(\theta)}[X]$; i.e. the subscript should use $p(\theta)$ instead of $p(\theta^*)$.

  • $\begingroup$ Thanks for the example. It's apparent that it doesn't hold in general, although it seems reasonable to expect it might for a not-too-narrow class of R.V.s. $\endgroup$ – broncoAbierto Feb 12 at 14:01
  • $\begingroup$ I accepted this answer as it shows that the general case does not hold, which was the main point of my question. Carlos Campos' answer provides interesting insight on the second point, though. $\endgroup$ – broncoAbierto Feb 13 at 8:50

Not an exhaustive answer, actually some hints for the second point:

For minimizing the mean (Using Leibniz rule): $$\left. \frac{d}{d \theta} \mathbb{E}_{X \sim p(\theta, x)} [x] \right\lvert_{\theta^*}= \int_{0^{+}}^{\infty} \left. \frac{\partial p(\theta, x)}{\partial \theta} \right\lvert_{\theta^*} x dx = 0 $$ And for minimizing the second moment: $$\left. \frac{d}{d \theta} \mathbb{E}_{X \sim p(\theta, x)} [x^2] \right\lvert_{\theta^*}= \int_{0^{+}}^{\infty} \left. \frac{\partial p(\theta, x)}{\partial \theta} \right\lvert_{\theta^*} x^2 dx = 0 $$ Combining both, we found next two sufficient conditions:

  • $\left. \frac{\partial p(\theta, x)}{\partial \theta} \right\lvert_{\theta^*} = 0 \quad \forall x$. For example, a uniform distribution $p(\theta,x) = \mathbb{U}_{[0,\theta]}(x)$ when $\theta \rightarrow 0$
  • $\left. \frac{\partial p(\theta, x)}{\partial \theta} \right\lvert_{\theta^*}$ is a function (not a probability distribution) whose first and second central moments are null. I can not find an example.

Union of both sufficient conditions should be the necessary condition.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.