Given a random variable $Y$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$.

If we take $n$ $IID$ samples from the distribution of $Y$, we can describe an **[Empirical Risk Minimization(ERM)][1]** procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $Y \sim N(0,\sigma^2)$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $E[L(Y,\hat{Y})]$ can be shown to be $\hat{Y} = \sigma^2$ using the fact that $e^{2Y}$ is lognormal with $E[e^{2Y}] = e^{2\sigma^2}$.

If we again apply **ERM** procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = -\frac{1}{2} ln \frac{n}{\sum_{i=1}^n e^{2Y_i}}$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of **ERM** am I violating?


  [1]: https://en.wikipedia.org/wiki/Empirical_risk_minimization