Average of Dependent Variables

Suppose $X_1, \ldots, X_n$ are dependent varibles with identical marginal distribution. Denote the common population mean as $\mu_0$. In this case, is $\frac{1}{n} \sum X_i$ a reasonably good estimator of $\mu_0$? I guess assymptotically it is still consistent. Is there any other suggestion?

• By "dependent variables" do you mean random variables that are not independent? If so, considering the case $X_1=X_2=\cdots=X_n=\cdots$ may clarify the situation for you.
– whuber
Commented Nov 6, 2014 at 17:41

If we have a collection of identically but not independently distributed random variables, then the sample mean is still an unbiased estimator of $\mu_0$. Its variance must now take into consideration the covariances between the random variables, so we cannot say something general about its finite sample efficiency.

As for consistency, the Weak Law of Large Numbers will hold if (but not "only if") correlation (and not necessarily dependence) vanishes asymptotically, in the sense of the sufficient condition associated with Markov's WLLN

$$\text{Var}(\bar X_n) \rightarrow 0 \Rightarrow \frac 1{n^2} \left (\sum_{i=1}^n\text{Var}(X_i) + {\sum \sum}_{i\neq j}\text{Cov}(X_i,X_j) \right)\rightarrow 0$$

$$\Rightarrow \frac 1{n} \text{Var}(X) + \frac 1{n^2}{\sum \sum}_{i\neq j}\text{Cov}(X_i,X_j) \rightarrow 0$$

The above requires that the individual variance is finite which makes also the first term go to zero. Then it it is the second term, the sum of covariances, that must go to zero.

Note that this condition does not imply some ordering along a natural index of the variables, like the time index: Although Markov's condition can be satisfied if the more intuitive notion "as variables are further apart in time, then they stop being correlated", this last condition is Bernstein's condition for his own version of WLLN. Markov's condition is more general, in that the scaled sum of covariances can go to zero in any kind of way.

Now, Markov's condition is designed for heterogeneous random variables. In our case the variables are identically distributed, so

$$\text{Cov}(X_i,X_j) = \rho_{ij}\text{Var}(X)$$

where $\rho_{ij}$ is the pairwise correlation coefficient, so the Markov's condition becomes here

$$\text{Var}(X)\frac 1{n^2}{\sum \sum}_{i\neq j}\rho_{ij} \rightarrow 0$$

Assume that the variables are pair-wise equicorrelated, $\rho_{ij}=\rho$ (by the way, equicorrelation is only possible if $\rho > -1/(n-1)$, otherwise the variance of the sample mean will be negative). Then we get

$$\text{Var}(X)\frac 1{n^2}{\sum \sum}_{i\neq j}\rho_{ij}= \text{Var}(X)\frac 1{n^2}(n^2-n)\rho$$

$$= \text{Var}(X)(1-1/n)\rho \rightarrow \text{Var}(X)\rho \neq 0$$

(which on the side means that to consider asymptotics of equicorrelated variables, we must have $\rho>0$).

So in such a case the Markov condition is not satisfied. Does this mean that the WLLN does not hold? Not necessarily, because the condition is only sufficient, not necessary. See this post (and links therein) for the case of consistent estimators that nevertheless have non-zero variance asymptotically (however demanding this may be to comprehend fully).

But of course there are cases when failure of the Markov condition signals inconsistency. Consider the case where the $X$'s are jointly normal, equicorrelated with $\rho>0$ , with marginal variance $\sigma^2<\infty$. Then we know, using also the previous results, that

$$\bar X_n \sim N\left(\mu_0, v_n\right),\;\; v_n = \sigma^2\big[\frac 1n + (1-1/n)\rho\big]$$

For consistency we examine ($\Phi$ is the standard normal CDF, $\epsilon >0$)

$$\lim_{n\rightarrow \infty}P\left(|\bar X_n-\mu_0|< \epsilon\right) = \lim_{n\rightarrow \infty}P\left(\frac{-\epsilon}{\sqrt v_n}<\frac{\bar X_n-\mu_0}{{\sqrt v_n}}< \frac{\epsilon}{\sqrt v_n}\right)$$

$$=\lim_{n\rightarrow \infty}\left (\Phi\left(\frac{\epsilon}{\sqrt v_n}\right)-\Phi\left(\frac{-\epsilon}{\sqrt v_n}\right)\right) = \lim_{n\rightarrow \infty}\left (2\Phi\left(\frac{\epsilon}{\sqrt v_n}\right)-1\right)$$

$$=2\Phi\left(\frac{\epsilon}{\sigma\sqrt \rho}\right)-1$$

and this last expression won't be equal to unity for all $\epsilon >0$ -and worse: the probability tends to zero as $\epsilon\rightarrow 0$, meaning that it is virtually unprobable for the distance between sample mean and true value to be close to zero. So in this case the sample mean is not a consistent estimator of $\mu_0$, although it is asymptotically unbiased, since

$$\bar X_n \xrightarrow{d} N\left(\mu_0, \sigma^2\rho\right)$$

But asymptotic unbiasedness is not the same as consistency.

• (+1) Is there a better phrasing than "virtually unprobable"? Commented Nov 7, 2014 at 8:55
• @Silverfish Any guesses? No problem in changing it, but my (non-native) English won't help me here. Commented Nov 7, 2014 at 18:43