5
$\begingroup$

Given a random variable $Y$ and the typical squared loss function:

$$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$

the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$.

If we take $n$ $IID$ samples from the distribution of $Y$, we can describe an Empirical Risk Minimization(ERM) procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n (Y_i - Y^*)^2$$

$$\implies \hat{Y} = \frac{1}{n}\sum_{i=1}^nY_i$$

$$E[\hat{Y}] = \mu$$

hence, it is consistent.

Now let's assume that $Y \sim N(0,\sigma^2)$ and our loss function is as follows:

$$L(Y,\hat{Y}) = e^{2(Y-\hat{Y})} - 2(Y-\hat{Y}) - 1$$

The mimizer for expected loss $E[L(Y,\hat{Y})]$ can be shown to be $\hat{Y} = \sigma^2$ using the fact that $e^{2Y}$ is lognormal with $E[e^{2Y}] = e^{2\sigma^2}$.

If we again apply ERM procedure:

$$\hat{Y} = \arg\min_{Y^*} \sum_{i=1}^n L(Y_i,Y^*)$$

$$\implies \hat{Y} = \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2Y_i} \right)$$

$$E[\hat{Y}] \not\rightarrow \sigma^2$$

I would like to understand why the procedure is not consistent in this case. Which assumptions of ERM am I violating?

$\endgroup$
4
  • $\begingroup$ By the strong law, $(1/n)\sum_{i=1}^n e^{2Y_i}$ converges a.s. to $\mathbb{E}[e^{2Y}]=e^{2\sigma^2}$. Hence, $(-1/2) \ln \left(n/\sum_{i=1}^n e^{2Y_i}\right)$ converges a.s. to $\sigma^2$, by continuity. $\endgroup$
    – Zen
    Dec 27, 2019 at 12:57
  • $\begingroup$ @Zen I ran your code. true_var <- 100; N <- 10^8; y <- rnorm(N, mean = 0, sd = sqrt(true_var)); (-0.5 * log(1 / mean(exp(2*y)))) [1] 47.79222 $\endgroup$ Dec 27, 2019 at 13:27
  • $\begingroup$ @Zen maybe it is overflowing? $\endgroup$ Dec 27, 2019 at 13:33
  • $\begingroup$ Doesn't look like it's overflowing. $\endgroup$
    – Zen
    Dec 27, 2019 at 13:42

1 Answer 1

3
$\begingroup$

The estimator is strongly consistent, since $$ \hat{\theta}_n = \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2Y_i} \right) \to \frac{1}{2} \, \ln \left( \mathbb{E}[e^{2Y}]\right) = \sigma^2, $$ almost surely.

Continuing the discussion started in the comments about the distribution of the estimator.

sim <- function(true_var, n, N) {
    y <- matrix(rnorm(N * n, mean = 0, sd = sqrt(true_var)), ncol = n)
    M <- apply(y, 1, max)
    dy <- sweep(y, 1, M, "-")
    est <- M + apply(dy, 1, function(row) 0.5 * log(mean(exp(2 * row))))
    hist(est, freq = TRUE, breaks = "FD", col = "cyan")
    abline(v = true_var, col = "red", lwd = 2)
}

set.seed(1234)

sim(true_var = 2, n = 10^4, N = 10^3)

enter image description here

sim(true_var = 10, n = 10^4, N = 10^3)

enter image description here

Note: in the function, I subtracted the maximum to prevent overflows: $$ \hat{\theta}_n = M + \frac{1}{2} \, \ln \left( \frac{1}{n} \sum_{i=1}^n e^{2(Y_i-M)}\right), $$ in which $M = \max_{1\leq i\leq n} Y_i$.

$\endgroup$
10
  • $\begingroup$ The strong law says that the estimator is consistent, but the sampling distribution of the estimator gets weird when we increase the value of the true variance. $\endgroup$
    – Zen
    Dec 27, 2019 at 13:45
  • $\begingroup$ I don't know much theory but isn't $E[ln f(x)] \neq ln E[f(x)]$, how does that continuity thing work? $\endgroup$ Dec 27, 2019 at 14:32
  • $\begingroup$ if $X_n$ converges a.s. to $X$, and $g$ is a continuous function, then $g(X_n)$ converges a.s. to $g(X)$. $\endgroup$
    – Zen
    Dec 27, 2019 at 14:35
  • 1
    $\begingroup$ It's possible for an estimator to be strongly consistent, but still be rubbish. See this for example: radfordneal.wordpress.com/2008/08/17/… $\endgroup$
    – Zen
    Dec 27, 2019 at 15:13
  • 1
    $\begingroup$ Billions isn't going to cut it for true_var = 10; the variance is on the order of $\exp\{20\}$, which is roughly $10^{8.7}$. You would need your actual sample size, not just the # of samples drawn, to be on the order of a billion to get the s.e. of the mean down below 1. $\endgroup$
    – jbowman
    Jan 14, 2020 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.