When is the optimizer of $\mathbb E[X]$ and $\mathbb E[X^2]$ the same? Consider a non-negative random variable $X\sim p(\theta)$, that is, following distribution $p$ parametrized by $\theta$.
Suppose we find a value of the parameters $\theta^*$ such that $$\mathbb E_{X\sim p(\theta)}[X]$$ is minimized. 


*

*Is $\mathbb E_{X\sim p(\theta)} [X^2]$ minimized for these
parameters as well?

*Otherwise, what might be sufficient and/or necessary conditions?

*Does it make a difference whether $X$ is a function of $\theta$?


*

*Edit: As kindly pointed out by whuber in the comments, $X$ is already a function of $\theta$, so assuming arbitrary variables/functions this last question is uninteresting.


 A: This is a partial answer actually. For the first subquestion, the optimizers are different, and an example should suffice I guess: 
Let $X$ be a RV such that it is $\sim\text{Bern(1/3)}$ with probability $\theta$, or equal to $1/2$ with probability $1-\theta$. $E[X]=\theta/3+(1-\theta)/2$, which is minimized when $\theta=1$. However, $E[X^2]=\theta/3+(1-\theta)/4$ is minimized when $\theta=0$. So, the optimizers for $E[X]$ and $E[X^2]$ yield different $\theta^*$.
Here, I don't have an answer for the sufficient/necessary conditions where they're equal; and I'm not sure if it can be found or not. 
For your last question, if $X$ is a function of $\theta$, then $\theta$ should be a random variable. But, we treat it as if it is an unknown constant and can be set to any value (based on its domain) depending on your objective function (e.g. minimizing $E[X]$). I can't make sense of the situation where $X=f(\theta)$, and we set $\theta$ to optimize $E[X]$.
Note: I think your notation should be $\mathbb E_{X\sim p(\theta)}[X]$; i.e. the subscript should use $p(\theta)$ instead of $p(\theta^*)$.
A: Not an exhaustive answer, actually some hints for the second point:
For minimizing the mean (Using Leibniz rule):
$$\left. \frac{d}{d \theta} \mathbb{E}_{X \sim p(\theta, x)} [x]  \right\lvert_{\theta^*}= \int_{0^{+}}^{\infty} \left. \frac{\partial p(\theta, x)}{\partial \theta} \right\lvert_{\theta^*} x dx  = 0 $$
And for minimizing the second moment:
$$\left. \frac{d}{d \theta} \mathbb{E}_{X \sim p(\theta, x)} [x^2]  \right\lvert_{\theta^*}=  \int_{0^{+}}^{\infty} \left. \frac{\partial p(\theta, x)}{\partial \theta} \right\lvert_{\theta^*} x^2 dx  = 0 $$
Combining both, we found next two sufficient conditions:


*

*$\left. \frac{\partial p(\theta, x)}{\partial \theta} \right\lvert_{\theta^*}  = 0 \quad \forall x$. For example, a uniform distribution $p(\theta,x) = \mathbb{U}_{[0,\theta]}(x)$ when $\theta \rightarrow 0$

*$\left. \frac{\partial p(\theta, x)}{\partial \theta} \right\lvert_{\theta^*}$ is a function (not a probability distribution) whose first and second central moments are null. I can not find an example.


Union of both sufficient conditions should be the necessary condition.
