2
$\begingroup$

I read about Method of Moments estimator (MOM) in Statistical Inference by Casella and Berger. In MOM description, I do not see the requirements that the sample should be iid. However, in the examples, there is an explicit assumption about the iid data.

As far as I understand, to compute MOM estimate, I need to equate the sample moments with the corresponding population moments and solve the resulting equations. It is not obvious to me that I need independent observations to compute that.

Could you please clarify if, in general, MOM estimator requires independent data points? What can go wrong if the observations are not independent?

$\endgroup$
3
$\begingroup$

Linearity of expectation doesn't rely on independence.

Consequently you don't require independence for $E(\bar{X})$ to equal $\mu$ (similarly for higher moments). So we just need that the expectation of $\overline{X^k}$ exists for it to equal $\mu^\prime_k$.

However if you want consistency (a property many people will expect in an estimator), you'll need that they're not "too dependent" (consider a sequence of perfectly correlated identically distributed variables, $X_1=X_2=X_3=...$ -- the variance of the average is the same as the variance of a single value -- you definitely don't have consistency there).

[There are, for example, versions of the weak law of large numbers for dependent variables with particular kinds of bounds on the covariances. (e.g. see this math.SE question).]


Now consider a couple of examples of sets of dependent variables to see that in some cases it will work okay and in others -- even with no variables being highly correlated -- it doesn't:

  1. Take $Z\sim (0,\sigma^2), X_i\sim (0,\sigma^2)\,,\quad i=1,2,...$ (where all variables are independent with the given mean and variance, and the $X_i$ are identically distributed)

    Now let $Y_i=\mu + kZ + \sqrt{1-k^2}\,X_i$ for some specific $0<k<1$.

    Then $E(Y_i)=\mu$ and $\text{Var}(Y_i) = \sigma^2$.

    \begin{eqnarray} \text{Cov}(Y_i,Y_j) &=& E[(Y_i-\mu)(Y_j-\mu)]\\ &=& E[(kZ + \sqrt{1-k^2}\,X_i)(kZ + \sqrt{1-k^2}\,X_j)]\\ &=&k^2E(Z^2)=k^2\sigma^2\,, \end{eqnarray}

    and the correlation between any pair of $Y$-variables is therefore $k$.

    $\text{Var}(Y_1+Y_2+...+Y_n) = (n\sigma^2 + n(n-1) k \sigma^2) $

    $\text{Var}(\bar{Y}) = \sigma^2(1/n + (n-1)/n k) > k\sigma^2 $

    Even though we have that $\bar{Y}$ is an unbiased estimator of $\mu$, this would not be particularly attractive because $\bar{Y}$ approaches $\mu+kZ$ -- even with a small common correlation, this estimate doesn't really improve -- once you add enough variables into the average to make the contributions of the $X$'s small, adding more variables in doesn't really help.

  2. Now consider another set of dependent variates:

    $Y_1=\mu+X_1\,,\quad$ (where the $X_i$ are defined as above)

    $Y_i=\mu+k(Y_{i-1}-\mu)+X_i\,,\quad i=2,3,...$ (again for some particular $0<k<1$).

    Now it turns out that we have that $E(\bar{Y})=\mu$ and $\sqrt{n}(\bar{Y}-\mu)$ has a finite variance (essentially because the correlations decrease geometrically in $|i-j|$, ultimately making it so the variance of the sum eventually grows only proportionally to $n$), so here $\bar{Y}$ makes more sense as an estimator of $\mu$ even though these variables are all correlated.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

As was pointed out in the comments, identically distributed data is required, otherwise, the estimate does not make sense.

As for the independence assumption, I guess that the main reason for it is the requirement imposed by the law of large numbers. Without iid, there is no guarantee that MOM estimate converges to the true mean.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Without the iid assumption, what would any estimate even mean? What would you be estimating if, for instance, each observation could be drawn from a different unknown distribution? $\endgroup$ – whuber Dec 31 '17 at 23:05
  • $\begingroup$ @whuber: "Without the iid assumption, what would any estimate even mean?": I don't think you mean fully iid: the independence assumption can be broken (i.e. sampling twins, etc.) and the MOM can still be very reasonable. $\endgroup$ – Cliff AB Mar 11 '18 at 16:11
  • $\begingroup$ @Cliff I did not intend to imply one must always have IID variables, but only that some such assumption is needed. If not IID, then exactly what will be assumed? MOM has no justification whatsoever (and may fail miserably) without some such assumption. $\endgroup$ – whuber Mar 11 '18 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.