6
$\begingroup$

Given a random variable $Y$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$.

If we take $n$ $IID$ samples from the distribution of $Y$, we can describe an Empirical Risk Minimization(ERM) procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $Y \sim N(0,\sigma^2)$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $E[L(Y,\hat{Y})]$ can be shown to be $\hat{Y} = \sigma^2$ using the fact that $e^{2Y}$ is lognormal with $E[e^{2Y}] = e^{2\sigma^2}$.

If we again apply ERM procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2Y_i} \right)$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of ERM am I violating?

$\endgroup$
  • $\begingroup$ By the strong law, $(1/n)\sum_{i=1}^n e^{2Y_i}$ converges a.s. to $\mathbb{E}[e^{2Y}]=e^{2\sigma^2}$. Hence, $(-1/2) \ln \left(n/\sum_{i=1}^n e^{2Y_i}\right)$ converges a.s. to $\sigma^2$, by continuity. $\endgroup$ – Zen Dec 27 '19 at 12:57
  • $\begingroup$ @Zen I ran your code. true_var <- 100; N <- 10^8; y <- rnorm(N, mean = 0, sd = sqrt(true_var)); (-0.5 * log(1 / mean(exp(2*y)))) [1] 47.79222 $\endgroup$ – Cagdas Ozgenc Dec 27 '19 at 13:27
  • $\begingroup$ @Zen maybe it is overflowing? $\endgroup$ – Cagdas Ozgenc Dec 27 '19 at 13:33
  • $\begingroup$ Doesn't look like it's overflowing. $\endgroup$ – Zen Dec 27 '19 at 13:42
3
$\begingroup$

The estimator is strongly consistent, since $$ \hat{\theta}_n = \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2Y_i} \right) \to \frac{1}{2} \, \ln \left( \mathbb{E}[e^{2Y}]\right) = \sigma^2, $$ almost surely.

Continuing the discussion started in the comments about the distribution of the estimator.

sim <- function(true_var, n, N) {
    y <- matrix(rnorm(N * n, mean = 0, sd = sqrt(true_var)), ncol = n)
    M <- apply(y, 1, max)
    dy <- sweep(y, 1, M, "-")
    est <- M + apply(dy, 1, function(row) 0.5 * log(mean(exp(2 * row))))
    hist(est, freq = TRUE, breaks = "FD", col = "cyan")
    abline(v = true_var, col = "red", lwd = 2)
}

set.seed(1234)

sim(true_var = 2, n = 10^4, N = 10^3)

enter image description here

sim(true_var = 10, n = 10^4, N = 10^3)

enter image description here

Note: in the function, I subtracted the maximum to prevent overflows: $$ \hat{\theta}_n = M + \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2(Y_i-M)}\right), $$ in which $M = \max_{1\leq i\leq n} Y_i$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The strong law says that the estimator is consistent, but the sampling distribution of the estimator gets weird when we increase the value of the true variance. $\endgroup$ – Zen Dec 27 '19 at 13:45
  • $\begingroup$ I don't know much theory but isn't $E[ln f(x)] \neq ln E[f(x)]$, how does that continuity thing work? $\endgroup$ – Cagdas Ozgenc Dec 27 '19 at 14:32
  • $\begingroup$ if $X_n$ converges a.s. to $X$, and $g$ is a continuous function, then $g(X_n)$ converges a.s. to $g(X)$. $\endgroup$ – Zen Dec 27 '19 at 14:35
  • 1
    $\begingroup$ It's possible for an estimator to be strongly consistent, but still be rubbish. See this for example: radfordneal.wordpress.com/2008/08/17/… $\endgroup$ – Zen Dec 27 '19 at 15:13
  • $\begingroup$ There is a long discussion in the comments here: stats.stackexchange.com/a/31038/20980. Is this one of those cases where a.s. convergence doesn't imply we are actually converging in the mean? $\endgroup$ – Cagdas Ozgenc Dec 28 '19 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.