1
$\begingroup$

I know that not all weakly consistent estimators exhibit MSE-consistency : https://stats.stackexchange.com/a/610835/397467.

Anyway, does increasing the sample size leads to a reduction in their mean squared error ?

Specifically, I am trying to find a formal mathematical argument to explain why models such as Logistic Regression (that are based on maximizing likelihood) gives better estimation of their coefficients when we train them on a large dataset. Maximum Likelihood Estimators are known to be consistent but I haven't found properties on their MSE.

$\endgroup$
2
  • $\begingroup$ pretty much for the same reason why CLT has a $.../n$ in the variance $\endgroup$
    – Alberto
    Commented Sep 28, 2023 at 13:09
  • $\begingroup$ yes i understand that for classical estimators as the empiric mean but in this case I don't get it. I think, in the described problem, it is more a convergence problem equivalent to "Convergence in probability implies a likely L^2 decrease" $\endgroup$
    – whn
    Commented Oct 2, 2023 at 14:37

1 Answer 1

2
$\begingroup$

Let's see how far we can go with this, using a bit naive/informal reasoning:

Define $(\hat \theta_n - \theta)^2 \equiv \Delta_n$. Then

$${\rm consistency}: \Pr(|\hat \theta_n - \theta| < \epsilon') \to 1 \implies \Pr(|\hat \theta_n - \theta|^2 > \epsilon'^2) \to 0,$$

$$\implies \Pr(\Delta_n \geq \epsilon) \to 0. \tag{1}$$

Also, $$MSE \equiv E(\hat \theta_n - \theta)^2 = E(\Delta_n). \tag{2}$$

For some $\epsilon >0$, decompose the expected value

$$E(\Delta_n) = E(\Delta_n \mid \Delta_n < \epsilon)\cdot\Pr(\Delta_n < \epsilon) + E(\Delta_n \mid \Delta_n \geq \epsilon)\cdot\Pr(\Delta_n \geq \epsilon).$$

We also have by basic properties of the expected value, $$E(\Delta_n \mid \Delta_n < \epsilon) \leq \epsilon,$$

and since $\Pr$ is bounded by unity $$MSE = E(\Delta_n) \leq \epsilon + E(\Delta_n \mid \Delta_n \geq \epsilon)\cdot\Pr(\Delta_n \geq \epsilon) \tag{3}$$

As sample size $n$ increases, we will eventually be able to choose smaller and smaller $\epsilon$. Also, we know from the consistency property that $\Pr(\Delta_n \geq \epsilon)$ will tend to zero. So the weak consistency property certainly pushes for the MSE to reduce in value.

More over, if $E(\Delta_n \mid \Delta_n \geq \epsilon)$ is bounded, the second term will certainly go to zero.

But, in the thread the OP links to, we see an example where in an analogous situation (with a mixture estimator), the corresponding part of the MSE stabilizes to a non-zero value.

In such a case also the MSE falls, although it does not reach zero. This and other such examples have certainly an artificial flavor, and their main purpose is to show why MSE-consistency is a stronger property.

But for most, if not all practical cases, we will see the MSE fall.

$\endgroup$
2
  • $\begingroup$ Thank you very much for your very delailed response ! However I don't understand why MSE should have certainly a decreasing value. I mean or $E(\Delta_n | \Delta_n > \epsilon )$ is bounded and then the estimator is MSE consistent or $E(\Delta_n | \Delta_n > \epsilon )$ diverge and then we cannot conclude anything on the decreasing of the MSE. I try to find something with successive decreasing values of epsilon but I don't succeed yet. $\endgroup$
    – whn
    Commented Oct 18, 2023 at 19:04
  • $\begingroup$ @whn The whole point is that to have a divergent $E(\Delta_n \mid \Delta_n \geq \epsilon)$, and in fact "so" divergent that it overcomes the plunge of $\Pr(\Delta_n \geq \epsilon)$ towards zero, is something that does not happen except in artificially constructed examples (caveat: all discussion here is subject to all these expected values existing of course). Think of what kind of distributions may $\Delta_n$ follow. $\endgroup$ Commented Oct 19, 2023 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.