Return to Question

Given a random variable $Y$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$.

If we take $n$ $IID$ samples from the distribution of $Y$, we can describe an Empirical Risk Minimization(ERM) procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $Y \sim N(0,\sigma^2)$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $E[L(Y,\hat{Y})]$ can be shown to be $\hat{Y} = \sigma^2$ using the fact that $e^{2Y}$ is lognormal with $E[e^{2Y}] = e^{2\sigma^2}$.

If we again apply ERM procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = -\frac{1}{2} ln \frac{n}{\sum_{i=1}^n e^{2Y_i}}$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of ERM am I violating?

Given a random variable $Y$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$.

If we take $n$ $IID$ samples from the distribution of $Y$, we can describe an Empirical Risk Minimization(ERM) procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $Y \sim N(0,\sigma^2)$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $E[L(Y,\hat{Y})]$ can be shown to be $\hat{Y} = \sigma^2$ using the fact that $e^{2Y}$ is lognormal with $E[e^{2Y}] = e^{2\sigma^2}$.

If we again apply ERM procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2Y_i} \right)$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of ERM am I violating?

Given a random variable $Y$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$.

If we take $n$ $IID$ samples from the distribution of $Y$, we can describe an Empirical Risk Minimization(ERM) procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $Y \sim N(0,\sigma^2)$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $E[L(Y,\hat{Y})]$ can be shown to be $\hat{Y} = \sigma^2$ using the fact that $e^{2Y}$ is lognormal with $E[e^{2Y}] = e^{2\sigma^2}$.

If we again apply ERM procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = -\frac{1}{2} ln \frac{n}{\sum_{i=1}^n e^{2Y_i}}$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of ERM am I violating?

Given a random variable $Y$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$.

If we take $n$ $IID$ samples from the distribution of $Y$, we can describe an Empirical Risk Minimization(ERM) procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $Y \sim N(0,\sigma^2)$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $E[L(Y,\hat{Y})]$ can be shown to be $\hat{Y} = \sigma^2$ using the fact that $e^{2Y}$ is lognormal with $E[e^{2Y}] = e^{2\sigma^2}$.

If we again apply ERM procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = -\frac{1}{2} ln \frac{n}{\sum_{i=1}^n e^{2Y_i}}$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of ERM am I violating?

Given a random variable $Y$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$.

If we take $n$ $IID$ samples from the distribution of $Y$, we can describe an Empirical Risk Minimization(ERM) procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $Y \sim N(0,\sigma^2)$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $E[L(Y,\hat{Y})]$ can be shown to be $\hat{Y} = \sigma^2$ using the fact that $e^{2Y}$ is lognormal with $E[e^{2Y}] = e^{2\sigma^2}$.

If we again apply ERM procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_i^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = -\frac{1}{2} ln \frac{n}{\sum_{i=1}^n e^{2Y_i}}$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of ERM am I violating?

Given a random variable $Y$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$.

If we take $n$ $IID$ samples from the distribution of $Y$, we can describe an Empirical Risk Minimization(ERM) procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $Y \sim N(0,\sigma^2)$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $E[L(Y,\hat{Y})]$ can be shown to be $\hat{Y} = \sigma^2$ using the fact that $e^{2Y}$ is lognormal with $E[e^{2Y}] = e^{2\sigma^2}$.

If we again apply ERM procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = -\frac{1}{2} ln \frac{n}{\sum_{i=1}^n e^{2Y_i}}$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of ERM am I violating?