# Inconsistent Empirical Risk Minimization procedure, but why?

Given a random variable $$Y$$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $$E[L(Y,\hat{Y})]$$ is know to be the mean, $$\hat{Y} = E[Y] = \mu$$.

If we take $$n$$ $$IID$$ samples from the distribution of $$Y$$, we can describe an Empirical Risk Minimization(ERM) procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $$Y \sim N(0,\sigma^2)$$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $$E[L(Y,\hat{Y})]$$ can be shown to be $$\hat{Y} = \sigma^2$$ using the fact that $$e^{2Y}$$ is lognormal with $$E[e^{2Y}] = e^{2\sigma^2}$$.

If we again apply ERM procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2Y_i} \right)$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of ERM am I violating?

• By the strong law, $(1/n)\sum_{i=1}^n e^{2Y_i}$ converges a.s. to $\mathbb{E}[e^{2Y}]=e^{2\sigma^2}$. Hence, $(-1/2) \ln \left(n/\sum_{i=1}^n e^{2Y_i}\right)$ converges a.s. to $\sigma^2$, by continuity.
– Zen
Dec 27, 2019 at 12:57
• @Zen I ran your code. true_var <- 100; N <- 10^8; y <- rnorm(N, mean = 0, sd = sqrt(true_var)); (-0.5 * log(1 / mean(exp(2*y)))) [1] 47.79222 Dec 27, 2019 at 13:27
• @Zen maybe it is overflowing? Dec 27, 2019 at 13:33
• Doesn't look like it's overflowing.
– Zen
Dec 27, 2019 at 13:42

The estimator is strongly consistent, since $$\hat{\theta}_n = \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2Y_i} \right) \to \frac{1}{2} \, \ln \left( \mathbb{E}[e^{2Y}]\right) = \sigma^2,$$ almost surely.

Continuing the discussion started in the comments about the distribution of the estimator.

sim <- function(true_var, n, N) {
y <- matrix(rnorm(N * n, mean = 0, sd = sqrt(true_var)), ncol = n)
M <- apply(y, 1, max)
dy <- sweep(y, 1, M, "-")
est <- M + apply(dy, 1, function(row) 0.5 * log(mean(exp(2 * row))))
hist(est, freq = TRUE, breaks = "FD", col = "cyan")
abline(v = true_var, col = "red", lwd = 2)
}

set.seed(1234)

sim(true_var = 2, n = 10^4, N = 10^3)


sim(true_var = 10, n = 10^4, N = 10^3)


Note: in the function, I subtracted the maximum to prevent overflows: $$\hat{\theta}_n = M + \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2(Y_i-M)}\right),$$ in which $$M = \max_{1\leq i\leq n} Y_i$$.

• The strong law says that the estimator is consistent, but the sampling distribution of the estimator gets weird when we increase the value of the true variance.
– Zen
Dec 27, 2019 at 13:45
• I don't know much theory but isn't $E[ln f(x)] \neq ln E[f(x)]$, how does that continuity thing work? Dec 27, 2019 at 14:32
• if $X_n$ converges a.s. to $X$, and $g$ is a continuous function, then $g(X_n)$ converges a.s. to $g(X)$.
– Zen
Dec 27, 2019 at 14:35
• It's possible for an estimator to be strongly consistent, but still be rubbish. See this for example: radfordneal.wordpress.com/2008/08/17/…
– Zen
Dec 27, 2019 at 15:13
• Billions isn't going to cut it for true_var = 10; the variance is on the order of $\exp\{20\}$, which is roughly $10^{8.7}$. You would need your actual sample size, not just the # of samples drawn, to be on the order of a billion to get the s.e. of the mean down below 1. Jan 14, 2020 at 16:20