9
$\begingroup$

Assume the following linear relationship: $Y_i = \beta_0 + \beta_1 X_i + u_i$, where $Y_i$ is the dependent variable, $X_i$ a single independent variable and $u_i$ the error term.

According to Stock & Watson (Introduction to Econometrics; Chapter 4), the third least squares assumption is that the fourth moments of $X_i$ and $u_i$ are non-zero and finite $(0<E(X_i^4)<\infty \text{ and } 0<E(u_i^4)<\infty)$.

I have three question:

  1. I do not fully understand the role of this assumption. Is OLS biased and inconsistent if this assumption does not hold or do we need this assumption for inference?

  2. Stock and Watson write "this assumption limits the probability of drawing an observation with extremely large values of $X_i$ or $u_i$." However, my intuition is that this assumption is extreme. Are we in trouble if we have large outliers (such that the fourth moments are large) but if these values are still finite? By the way: What is the underlying definition an outlier?

  3. Can we reformulate this as follows: "The kurtosis of $X_i$ and $u_i$ are nonzero and finite?"

$\endgroup$
  • $\begingroup$ Unfortunately I can't write a fully fledged answer now but to answer you question: 1, OLS consistency works regardless. 2, no clear definition of outliers exists, but OLS works fine in large sample in the presence of outliers. 3, for the life of me I can't think of an example where that would not be true, but someone could prove me wrong so no guarantees $\endgroup$ – Repmat Oct 31 '16 at 20:19
  • 5
    $\begingroup$ I dispute "but OLS works fine in large sample in the presence of outliers" ... take a large enough outlier in x-space (i.e. an influential observation) and a single point can force the LS fit to go through it; if it's also an outlier in the Y-direction, your line still will go though that one point, no matter how extreme it is. $\endgroup$ – Glen_b Nov 1 '16 at 2:08
  • 2
    $\begingroup$ Outliers are easy to define. They are observations inconsistent with the pattern of the bulk of the data. As the example by Glen_b shows, such point have undue influence on the fit, at the limit outweighing all other observation in the dataset, leading to highly biased estimates. $\endgroup$ – user603 Nov 1 '16 at 7:57
  • 1
    $\begingroup$ @user603 Sure... and so what... I have yet to encounter a program/script that automatically detects outliers and does so in clear way that we all agree is the right way... so while I agree with your sentiment, it doesnt help OP $\endgroup$ – Repmat Nov 2 '16 at 12:19
  • $\begingroup$ @Repmat: please re-read the OP's question. My comment directly answers one of the sentences in there that is punctuated by a question mark. $\endgroup$ – user603 Nov 2 '16 at 12:25
9
$\begingroup$

You do not need assumptions on the 4th moments for consistency of the OLS estimator, but you do need assumptions on higher moments of $x$ and $\epsilon$ for asymptotic normality and to consistently estimate what the asymptotic covariance matrix is.

In some sense though, that is a mathematical, technical point, not a practical point. For OLS to work well in finite samples in some sense requires more than the minimal assumptions necessary to achieve asymptotic consistency or normality as $n \rightarrow \infty$.

Sufficient conditions for consistency:

If you have regression equation: $$ y_i = \mathbf{x}_i' \boldsymbol{\beta} + \epsilon_i $$

The OLS estimator $\hat{\mathbf{b}}$ can be written as: $$ \hat{\mathbf{b}} = \boldsymbol{\beta} + \left( \frac{X'X}{n}\right)^{-1}\left(\frac{X'\boldsymbol{\epsilon}}{n} \right)$$

For consistency, you need to be able to apply Kolmogorov's Law of Large Numbers or, in the case of time-series with serial dependence, something like the Ergodic Theorem of Karlin and Taylor so that:

$$ \frac{1}{n} X'X \xrightarrow{p} \mathrm{E}[\mathbf{x}_i\mathbf{x}_i'] \quad \quad \quad \frac{1}{n} X'\boldsymbol{\epsilon} \xrightarrow{p} \mathrm{E}\left[\mathbf{x}_i' \epsilon_i\right] $$

Other assumptions needed are:

  • $\mathrm{E}[\mathbf{x}_i\mathbf{x}_i']$ is full rank and hence the matrix is invertible.
  • Regressors are predetermined or strictly exogenous so that $\mathrm{E}\left[\mathbf{x}_i \epsilon_i\right] = \mathbf{0}$.

Then $\left( \frac{X'X}{n}\right)^{-1}\left(\frac{X'\boldsymbol{\epsilon}}{n} \right) \xrightarrow{p} \mathbf{0}$ and you get $\hat{\mathbf{b}} \xrightarrow{p} \boldsymbol{\beta}$

If you want the central limit theorem to apply then you need assumptions on higher moments, for example, $\mathrm{E}[\mathbf{g}_i\mathbf{g}_i']$ where $\mathbf{g_i} = \mathbf{x}_i \epsilon_i$. The central limit theorem is what gives you asymptotic normality of $\hat{\mathbf{b}}$ and allows you to talk about standard errors. For the second moment $\mathrm{E}[\mathbf{g}_i\mathbf{g}_i']$ to exist, you need the 4th moments of $x$ and $\epsilon$ to exist. You want to argue that $\sqrt{n}\left(\frac{1}{n} \sum_i \mathbf{x}_i' \epsilon_i \right) \xrightarrow{d} \mathcal{N}\left( 0, \Sigma \right)$ where $\Sigma = \mathrm{E}\left[\mathbf{x}_i\mathbf{x}_i'\epsilon_i^2 \right]$. For this to work, $\Sigma$ has to be finite.

A nice discussion (which motivated this post) is given in Hayashi's Econometrics. (See also p. 149 for 4th moments and estimating the covariance matrix.)

Discussion:

These requirements on 4th moments is probably a technical point rather than a practical point. You're probably not going to encounter pathological distributions where this is a problem in everyday data? It's for more commonf or other assumptions of OLS to go awry.

A different question, undoubtedly answered elsewhere on Stackexchange, is how large of a sample you need for finite samples to get close to the asymptotic results. There's some sense in which fantastic outliers lead to slow convergence. For example, try estimating the mean of a lognormal distribution with really high variance. The sample mean is a consistent, unbiased estimator of the population mean, but in that log-normal case with crazy excess kurtosis etc... (follow link), finite sample results are really quite off.

Finite vs. infinite is a hugely important distinction in mathematics. That's not the problem you encounter in everyday statistics. Practical problems are more in the small vs. big category. Is the variance, kurtosis etc... small enough so that I can achieve reasonable estimates given my sample size?

Pathological example where OLS estimator is consistent but not asymptotically normal

Consider:

$$ y_i = b x_i + \epsilon_i$$ Where $x_i \sim \mathcal{N}(0,1)$ but $\epsilon_i$ is drawn from a t-distribution with 2 degrees of freedom thus $\mathrm{Var}(\epsilon_i) = \infty$. The OLS estimate converges in probability to $b$ but the sample distribution for the OLS estimate $\hat{b}$ is not normally distributed. Below is the empirical distribution for $\hat{b}$ based upon 10000 simulations of a regression with 10000 observations. QQPlot for estimator (does not converge in distribution to normal)

The distribution of $\hat{b}$ isn't normal, the tails are too heavy. But if you increase the degrees of freedom to 3 so that the second moment of $\epsilon_i$ exists then the central limit applies and you get: QQPlot for estimator (converges in distribution to normal)

Code to generate it:

beta = [-4; 3.7];
n = 1e5;    
n_sim = 10000;    
for s=1:n_sim
    X = [ones(n, 1), randn(n, 1)];  
    u  = trnd(2,n,1) / 100;
    y = X * beta + u;

    b(:,s) = X \ y;
end
b = b';
qqplot(b(:,2));
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Nice answer. But the following really depends on the context: You're not going to encounter pathological distributions with non-existent 4th moments in everyday data. Financial data (log-returns on financial assets) typically are as heavy-tailed as not to have a finite 4th moment. So the concern over the 4th moment is very real there. (You could probably add this as a parenthetical counterexample to your claim.) Also, a question: in your example, why does $t(3)$ yield asymptotic normality despite not having a finite 4th moment? $\endgroup$ – Richard Hardy Nov 2 '16 at 10:02
  • 1
    $\begingroup$ @RichardHardy You want $\sqrt{n}\left( \frac{1}{n} \sum_i \mathbf{x}_i \epsilon_i \right) \xrightarrow{d} \mathcal{N}\left( \mathbf{0}, \Sigma \right)$ where $\Sigma = \mathrm{E}[\mathbf{x}_i\mathbf{x}_i'\epsilon_i^2]$. You need that 4th moment $\Sigma$ to exist, and $\Sigma$ is basically a second moment in $\epsilon_i$ when $\epsilon_i^2$ is uncorrelated with $\mathbf{x}_i\mathbf{x}_i'$. $\endgroup$ – Matthew Gunn Nov 2 '16 at 12:37
6
$\begingroup$
  1. This is a sufficient assumption, but not a minimal one [1]. OLS is not biased under these conditions, it is just inconsistent. The asymptotic properties of OLS break down when $X$ can have extremely large influence and/or if you can obtain extremely large residuals. You may not have encountered a formal presentation of the Lindeberg Feller central limit theorem, but that is what they are addressing here with the fourth moment conditions, and the Lindeberg condition tells us basically the same thing: no overlarge influence points, no overlarge high leverage points [2].

  2. These theoretical underpinnings of statistics cause a lot of confusion when boiled down for practical applications. There is no definition of an outlier, it is an intuitive concept. To understand it roughly, the observation would have to be a high leverage point or high influence point, e.g. one for which the deletion diagnostic (DF beta) is very large, or for which the Mahalanobis distance in the predictors is large (in univariate stats that's just a Z score). But let's return to practical matters: if I conduct a random survey of people and their household income, and out of 100 people, 1 of the persons I sample is a millionaire, my best guess is that millionaires are representative of 1% of the population. In a biostatistcs lecture, these principals are discussed and emphasized that any diagnostic tool is essentially exploratory[3]. A nice point is made here: when exploratory statistics uncover an outlier, the "result" of such an analysis is not "the analysis which excludes the outlier is the one I believe", it is, "removing one point completely changed my analysis."

  3. Kurtosis is a scaled quantity which depends upon the second moment of a distribution, but the assumption of finite, non-zero variance for these values is tacit since it is impossible for this property to hold in the fourth moment but not in the second. So basically yes, but overall I have never inspected either kurtosis or fourth moments. I don't find them to be a practical or intuitive measure. In this day when a histogram or scatter plot is produced by the snap of one's fingers, it behooves us to use qualitative graphical diagnostic statistics, by inspecting these plots.

[1] https://math.stackexchange.com/questions/79773/how-does-one-prove-that-lindeberg-condition-is-satisfied

[2] http://projecteuclid.org/download/pdf_1/euclid.ss/1177013818

[3] http://faculty.washington.edu/semerson/b517_2012/b517L03-2012-10-03/b517L03-2012-10-03.html

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ As has been pointed out before, ones's intuition about outliers breaks down when there are more than one of them. They will not necessarily stand out in a DF beta plot or have large z-scores because these statistics themselves can be swayed by outliers. As we discussed before, outliers, if left unchecked, will produced biased coefficients unless you remove them or use an estimation technique robust to them. $\endgroup$ – user603 Nov 1 '16 at 10:55
  • 1
    $\begingroup$ I think more generally, when expressing opinions, your answers would gain by including pointers to the relevant literature so that the OP knows which one of these opinion are widely held. $\endgroup$ – user603 Nov 1 '16 at 10:55
  • $\begingroup$ @user603 To your first comment, I have not pointed to DFbetas (or any diagnostic tool) as an exclusive method for identifying outliers, but certainly a useful one. When performing semi-parametric inference (mean model correct) outliers do NOT bias LS models, Can you produce a reference or even an example in any case other than non-parametric LS? Your second comment is a good one, and I'll take the next several moments to supply citations. $\endgroup$ – AdamO Nov 1 '16 at 16:47
  • $\begingroup$ Your statement, "OLS is not biased under these conditions, it is just inconsistent" is not correct. The higher moments are needed for asymptotic normality. They are not needed for consistency in IID samples where Kolmogorov Law of Large Numbers applies. $\endgroup$ – Matthew Gunn Nov 4 '16 at 0:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.