When to use Student's or Normal distribution in linear regression?

Question

I am looking at some problems, and in some, to test the coefficients, sometimes I see people using Student's distribution, and sometimes I see Normal distribution. What is the rule?

Answer 1

The normal distribution is the large sample distribution in many meaningful statistical problems that involve some version of the Central Limit Theorem: you have (approximately) independent pieces of information that are being added up to arrive at the answer. If parameter estimates are asymptotically normal, their functions will also be asymptotically normal (in regular cases).

On the other hand, the Student $t$ distribution is derived under more restrictive conditions of i.i.d. normal regression errors. If you can buy this assumption, you can buy the $t$-distribution being used for testing hypothesis in linear regression. The use of this distribution provides wider confidence intervals than the use of the normal distribution. The substantive meaning of that is that in small samples, you need to estimate your measure of uncertainty, the regression mean squared error, or the standard deviation of residuals, $\sigma$. (In large samples, you kinda have as much information as if you knew it, so the $t$-distribution degenerates to the normal distribution.)

There are some occasions in linear regression, even with finite samples, where the Student distribution cannot be justified. They are related to violations of the second order conditions on regression errors; namely, that they are (1) constant variance, and (2) independent. If these assumptions are violated, and you correct your standard errors using Eicker/White estimator for heteroskedastic, but independent residuals; or Newey-West estimator for serially correlated errors, or clustered standard errors for cluster-correlated data, there is no way you can pull a reasonable justification for Student distribution. However, by employing an appropriate version of asymptotic normality argument (traingular arrays and such), you can justify the normal approximation (although you should have in mind that your confidence intervals would very likely be too narrow).

Answer 2

I like the representation of the student t distribution as a mixture of a normal distribution and a gamma distribution:

$$Student(x|\mu,\sigma^2,\nu)=\int_{0}^{\infty}Normal\left(x|\mu,\frac{\sigma^2}{\rho}\right)Gamma\left(\rho|\frac{\nu}{2},\frac{\nu}{2}\right)d\rho$$

Note that the mean of the gamma distribution is $E[\rho|\nu]=1$ and the variance of this distribution is $V[\rho|\nu]=\frac{2}{\nu}$. So we can view the t-distribution as generalising the constant variance assumption to a "similar" variance assumption. $\nu$ basically controls how similar we allow the variances to be. You also view this as "random weighted" regression, for we can use the above integral as a "hidden variable" representation as follows:

$$y_i=\mu_i+\frac{e_i}{\sqrt{\rho_i}}$$

Where $e_i\sim N(0,\sigma^2)$ and $\rho_i\sim Gamma\left(\frac{\nu}{2},\frac{\nu}{2}\right)$ all variables independent. In fact this is basically just the definition of the t-distribution, as $Gamma\left(\frac{\nu}{2},\frac{\nu}{2}\right)\sim \frac{1}{\nu}\chi^2_\nu$

You can see why this result makes the student t distribution "robust" compared to the normal because a large error $y_i-\mu_i$ can occur due to a large value of $\sigma^2$ or due to a small value of $\rho_i$. Now becuase $\sigma^2$ is common to all observations, but $\rho_i$ is specific to the ith one, the general "common sense" thing to conclude is that outliers give evidence for small $\rho_i$. Additionally, if you were to do linear regression $\mu_i=x_i^T\beta$, you will find that $\rho_i$ is the weight for the ith observation, assuming that $\rho_i$ is known.:

$$\hat{\beta}=(\sum_i\rho_ix_ix_i^T)^{-1}(\sum_i\rho_ix_iy_i)$$

So an outlier constitutes evidence for small $\rho_i$ which means the ith observation gets less weight. Additionally, an small "outlier" - an observation which is predicted/fitted much better than the rest - constitutes evidence for large $\rho_i$. Hence this observation will be given more weight in the regression. This is in line with what one would intuitively do with an outlier or a good data point.

Note that there is no "rule" for deciding these things, although mine and others response to this question may be useful for finding some tests you can do along the finite variance path (student t is infinite variance for degrees of freedom less than or equal to two).