# When to use Student's or Normal distribution in linear regression?

I am looking at some problems, and in some, to test the coefficients, sometimes I see people using Student's distribution, and sometimes I see Normal distribution. What is the rule?

-
This is not an answer, but do note that the $t$-distribution approaches the normal distribution as the degrees-of-freedom parameter $\nu$ grows larger. Past $\nu \geq 30$, there is no appreciable difference, particularly in most hypothesis-testing frameworks. The limiting behavior is "from above" in the sense that the if $T \sim t_{\nu}$ and $Z \sim \mathcal{N}(0,1)$, then $|T|$ is stochastically larger than $|Z|$. –  cardinal Aug 10 '11 at 16:57

The normal distribution is the large sample distribution in many meaningful statistical problems that involve some version of the Central Limit Theorem: you have (approximately) independent pieces of information that are being added up to arrive at the answer. If parameter estimates are asymptotically normal, their functions will also be asymptotically normal (in regular cases).

On the other hand, the Student $t$ distribution is derived under more restrictive conditions of i.i.d. normal regression errors. If you can buy this assumption, you can buy the $t$-distribution being used for testing hypothesis in linear regression. The use of this distribution provides wider confidence intervals than the use of the normal distribution. The substantive meaning of that is that in small samples, you need to estimate your measure of uncertainty, the regression mean squared error, or the standard deviation of residuals, $\sigma$. (In large samples, you kinda have as much information as if you knew it, so the $t$-distribution degenerates to the normal distribution.)

There are some occasions in linear regression, even with finite samples, where the Student distribution cannot be justified. They are related to violations of the second order conditions on regression errors; namely, that they are (1) constant variance, and (2) independent. If these assumptions are violated, and you correct your standard errors using Eicker/White estimator for heteroskedastic, but independent residuals; or Newey-West estimator for serially correlated errors, or clustered standard errors for cluster-correlated data, there is no way you can pull a reasonable justification for Student distribution. However, by employing an appropriate version of asymptotic normality argument (traingular arrays and such), you can justify the normal approximation (although you should have in mind that your confidence intervals would very likely be too narrow).

-
(+1) I love the implication, in the opening of the third paragraph, that linear regression is done with infinite (non-"finite") samples! –  whuber Aug 10 '11 at 17:11
@whuber: :) In my books, if it is normal, it must be relying on CLT or something asymptotic. Otherwise, it makes as much sense as this. –  StasK Aug 10 '11 at 17:49

I like the representation of the student t distribution as a mixture of a normal distribution and a gamma distribution:

$$Student(x|\mu,\sigma^2,\nu)=\int_{0}^{\infty}Normal\left(x|\mu,\frac{\sigma^2}{\rho}\right)Gamma\left(\rho|\frac{\nu}{2},\frac{\nu}{2}\right)d\rho$$

Note that the mean of the gamma distribution is $E[\rho|\nu]=1$ and the variance of this distribution is $V[\rho|\nu]=\frac{2}{\nu}$. So we can view the t-distribution as generalising the constant variance assumption to a "similar" variance assumption. $\nu$ basically controls how similar we allow the variances to be. You also view this as "random weighted" regression, for we can use the above integral as a "hidden variable" representation as follows:

$$y_i=\mu_i+\frac{e_i}{\sqrt{\rho_i}}$$

Where $e_i\sim N(0,\sigma^2)$ and $\rho_i\sim Gamma\left(\frac{\nu}{2},\frac{\nu}{2}\right)$ all variables independent. In fact this is basically just the definition of the t-distribution, as $Gamma\left(\frac{\nu}{2},\frac{\nu}{2}\right)\sim \frac{1}{\nu}\chi^2_\nu$

You can see why this result makes the student t distribution "robust" compared to the normal because a large error $y_i-\mu_i$ can occur due to a large value of $\sigma^2$ or due to a small value of $\rho_i$. Now becuase $\sigma^2$ is common to all observations, but $\rho_i$ is specific to the ith one, the general "common sense" thing to conclude is that outliers give evidence for small $\rho_i$. Additionally, if you were to do linear regression $\mu_i=x_i^T\beta$, you will find that $\rho_i$ is the weight for the ith observation, assuming that $\rho_i$ is known.:

$$\hat{\beta}=(\sum_i\rho_ix_ix_i^T)^{-1}(\sum_i\rho_ix_iy_i)$$

So an outlier constitutes evidence for small $\rho_i$ which means the ith observation gets less weight. Additionally, an small "outlier" - an observation which is predicted/fitted much better than the rest - constitutes evidence for large $\rho_i$. Hence this observation will be given more weight in the regression. This is in line with what one would intuitively do with an outlier or a good data point.

Note that there is no "rule" for deciding these things, although mine and others response to this question may be useful for finding some tests you can do along the finite variance path (student t is infinite variance for degrees of freedom less than or equal to two).

-
+1: this looks right, but I don't think you should say a mixture of a normal and a gamma distribution, but rather a normal-gamma–normal compound distribution and motivate this construction by saying that the normal-gamma distribution is the conjugate prior of the normal distribution (parametrized by mean and precision). –  Neil G Mar 28 '12 at 12:40
Yeah, point taken about the mixture - although I can't think of a non-clumsy way to correct it right now. Note that this form is not unique to conjugate distributions - for example if we replace the gamma pdf with an inverted exponential pdf, we get the laplace distribution. This leads to "least absolute deviations" instead of least squares as a form of robustifying the normal distribution. Other distributions would lead to other "robustifications" - perhaps not as analytically pretty as student t though. –  probabilityislogic Mar 28 '12 at 13:04