2
$\begingroup$

My question is about t-test values for linear regression coefficients using OLS and MLE. There are a few related posts on this website (here, and here ), and I could not find the exact answer.

The following are a few starting points to set the context:

(1) We know that OLS and MLE will generate the same estimates for regression coefficients(e.g., $\hat{\beta}_0$, $\hat{\beta}_1$).

(2) We know that linear regression coefficients estimated by OLS do not assume normal distribution for $\epsilon$. Thus, if we want to get a p-value for estimated coefficients, typically we assume normal distribution for $\epsilon$, same as MLE.

(3) Below is from the textbook of Bain and Engelhardt's Intro. to Prob. and Math. Stat. (p. 514). The $T_0$ and $T_1$ are for intercept and slope t tests in MLE for simple linear regression. We can see that it cancels out the population $\sigma^2$ and only keeps the unbiased estimate $\tilde{\sigma}$ in the t-statistic. We know that the estimate of $\sigma^2$ in OLS is also unbiased $\tilde{\sigma}$.

My question is: Are t-statistic formulas and values for linear regression coefficients the same across OLS and MLE?

I believe so. If it is not true, can you point out where I make mistakes in this reasoning process? Thank you so much. I look forward to your insights, suggestions, and comments.

(Note that, let's just assume $\sigma^2$ is unkown, and we will just use t-test rather than standard normal, to limit the scope of the discussion here. I asked a related question here, but not the same question. I did not have space in that question to discuss this, and thus I am asking it as a new question. Plus, this is a different question. You might point out that there are other statistics (not just t-test) to test regression coefficients in MLE. However, let's just stick to t-test in this question, to limit the scope of this question. Thank you so much.)

enter image description here

Added comments - Part 1:

(1) You might have questions regarding how to derive estimated $\sigma^2$ in MLE. If so, please refer to this PDF from Ryan Adams. The short answer is that, in MLE, $\hat{\sigma}^2 =\frac{\sum_{i=1}^n (y_i – \hat{\beta}_0-\hat{\beta}_1 x_i)^2}{n}$ is a biased estimate of $\sigma^2$. However, note that, it does not really matter here though, since the t-test shown above uses unbiased $\tilde{\sigma}$.

(2) I took a photo on the section of $\sigma^2$ in OLS (p.502). It provides the context of how $\sigma^2$ links with $\tilde{\sigma}^2$.

enter image description here

Added comments - Part 2 Added on July 21, 2023:

The picture above provides how $\tilde{\sigma}^2$ is derived in OLS. There are questions regarding how $\tilde{\sigma}^2$ is derived in MLE then.

Note that, as mentioned in the picture above, $\tilde{\sigma}^2$ is the same across OLS and MLE. Below are Theorem 15.3.4, 5, 6 about this from Bain and Engelhardt's Intro. to Prob. and Math. Stat. (p. 510-514).

enter image description here enter image description here enter image description here

$\endgroup$
37
  • $\begingroup$ Welcome to Cross Validated! Maximum likelihood estimation according to what likelihood? $\endgroup$
    – Dave
    Jul 19, 2023 at 14:30
  • 1
    $\begingroup$ Hi Will: it's an interesting question but let me throw one back at you. When one solves for the MLE of $\beta$, how does one obtain the MLE for $\sigma^2$. That's important because, how one does that, determines whether the 2 parameter estimates of $\sigma^2$ from the two different methodologies will return the same values ? If they are the same, then the t-statistics one constructs will be the same because all the parameter estimates are the same. If they are not the same, then the t-statistics one constructs will be slightly different and how different will depend on how large $n$ is. $\endgroup$
    – mlofton
    Jul 19, 2023 at 17:16
  • 1
    $\begingroup$ @whuber Thank you for stopping by! You are right in a sense that $\tilde{\sigma}^2$ is used in OLS. $\tilde{\sigma}^2 =\frac{SSE}{n-2} =\frac{\sum_{i=1}^n (y_i – \hat{\beta}_0-\hat{\beta}_1 x_i)^2}{n-2}$ is an ubiased estimate of $\sigma^2$. Further, estimated $\sigma^2$ from MLE is biased. However, in MLE, unbiased estimate of $\sigma^2$ is also $\tilde{\sigma}^2$. Thus, the $V \sim \chi^2(n-2)$ shown in the pic. is independent from the fact that $\tilde{\sigma}^2$ is from OLS or MLE, since $\tilde{\sigma}^2$ is the same across OLS and MLE. $\endgroup$
    – Will
    Jul 19, 2023 at 18:49
  • 1
    $\begingroup$ Therefore, since these respective estimates are used in the calculation of the respective t-statistics, you won't get the same EXACT result for the t-statistics. But, for larger $n$ it won't matter in the sense that the difference is negligible. $\endgroup$
    – mlofton
    Jul 19, 2023 at 20:03
  • 1
    $\begingroup$ This means that the $\sigma^2$ estimate from the MLE approach will always be different from that of the OLS approach. $\endgroup$
    – mlofton
    Jul 19, 2023 at 20:12

2 Answers 2

1
$\begingroup$

For given observations $Y$ and regression matrix $\mathbf{X}$, if we assume independent normal distributed observations with equal standard deviation.

$$Y|X \sim \mathcal{N}(X \beta,\sigma)$$

then the coefficients of the linear model are computed as

  • $\hat{\beta} = (\mathbf{X^TX})^{-1}\mathbf{X^T}Y$ for ordinary least squares estimation.
  • $\tilde{\beta} = (\mathbf{X^TX})^{-1}\mathbf{X^T}Y$ for maximum likelihood estimation.

The estimates are the same.

Also the sampling distribution of the estimate is the same and with the same distributional assumptions the inference with confidence intervals or p-values should be the same.

Differences may occur when different approaches are used to the computation of confidence intervals or p-values. For example computing one-sided or two-sided p-values. But, this is not in principle a difference between ordinary least squares and maximum likelihood estimation.


When we do not make the assumption of independent normal distributed observations, then the two methods will be different (see:Why are the Least-Squares and Maximum-Likelihood methods of regression not equivalent when the errors are not normally distributed?). The estimates can be different because the maximum likelihood does not need to coincidence with minimizing least squares.

In a special case we can have the same estimates but different p-values and confidence intervals. This occurs when the maximum likelihood estimation is computed with the assumption of normal distribution, but the ordinary least square method with a different assumption. Then the estimates are both determined by minimizing the sum of squared residuals, but the assumptions about the sampling distribution of the estimate is different.

$\endgroup$
2
  • $\begingroup$ Sextus Empiricus: Thank you for the feedback. I tend to agree with you. People in the comment section mentioned that the estimated $\sigma^2$ differ between OLS ($ \hat{\sigma}^2_{OLS}=\frac{\sum_{i=1}^n (y_i - \hat{\beta}_0-\hat{\beta}_1 x_i)^2}{n-2}$) and MLE ($ \hat{\sigma}^2_{MLE} =\frac{\sum_{i=1}^n (y_i - \hat{\beta}_0-\hat{\beta}_1 x_i)^2}{n}$). However, when constructing the $V \sim \chi^2(n-2)$ for t-statistic: $\hat{\sigma}^2_{OLS} * (n-2)/\sigma^2=\hat{\sigma}^2_{MLE} * (n)/\sigma^2$. Thus, t-statistic are the same, eventually. $\endgroup$
    – Will
    Jul 23, 2023 at 19:36
  • $\begingroup$ Note that, $V$ refer to the $V$ mentioned in the Theorem 15.3.6. Further, since $Z_0$ and $Z_1$ in Theorem 15.3.6 are the same across OLS and MLE as well. Thus, t-statistic are the same. Let's see if others have comments. Otherwise, I would accept your answer. $\endgroup$
    – Will
    Jul 23, 2023 at 19:41
0
$\begingroup$

let me again try to describe the distribution for $\hat{\beta}_1$ in OLS.

First, let's assume that the proof of
$$\frac{\hat{\beta}_1 - \beta_1}{\sqrt{\sigma^2/S_{xx}}} ~\sim~ N(0,1) $$ has already been provided and that $S_{xx}$ has its usual interpretation.

We need to estimate $\sigma^2$ and will denote $\tilde{\sigma}^2$ as the estimator: $$ \tilde{\sigma}^2 = \frac{1}{n-2} \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2. $$

Also, the following two statements would be proven in any decent math-stats textbook. We will just assume that they are true: \begin{equation*} \frac{(n-2)\tilde{\sigma}^2}{\sigma^2} ~\sim~ \chi^2_{n-2} \mathrm{~~~and~~~} \frac{N(0,1)}{\sqrt{{\chi^2_d}/{d}}} ~\sim~ t(d). \end{equation*}

Hence, by taking the normal random variable and dividing it by the square root of the $\chi^2(n-2)$ divided by $(n-2)$ random variable : $$ \frac{\hat{\beta}_1 - \beta_1}{\sqrt{\sigma^2/S_{xx}}} \Big/ \sqrt{\frac{(n-2)\mathrm{\tilde{\sigma}^2}}{\sigma^2} \Big/ (n-2)} ~\sim~ t(n-2), $$

Fortunately, after a lot of algebra, the expression can be reduced to: $$ \frac{\hat{\beta}_1 - \beta_1}{\sqrt{\tilde{\sigma}^2/S_{xx}}} = \frac{\hat{\beta}_1 - \beta_1}{\mathrm{SE}(\hat{\beta}_1)} $$

This means that the reduced expression also has to be distributed as $t(n-2)$. $$ \boxed{ \frac{\hat{\beta}_1 - \beta_1}{\sqrt{\mathrm{\tilde{\sigma}^2}/S_{xx}}} = \frac{\hat{\beta}_1 - \beta_1}{\mathrm{SE}(\hat{\beta}_1)} ~\sim t(n-2) } $$

Notice that the final expression only has $\hat{\beta}_1$ and $\tilde\sigma^2$ as inputs. It is called the t-statistic.

Now, to your original question. Are the t-statistics generated by the ML and OLS procedures the same. The t-statistic has only two inputs. One is an estimate of $\beta$ and the other is the estimate of $\sigma^2$. So, to know whether ML and OLS generate the same t-statistics we just have to know whether the two inputs are the same in both procedures. They are not because the ML procedure uses $n$ as the divisor when calculating $\tilde{\sigma}^2$. The OLS procedure on the other hand uses $(n-2)$. Therefore, the t-statistics generated by the two procedures will not be the same.

I hope this helps. It's pretty similar to what I deleted earlier, except that I was considering a non-regression framework ( hypothesis testing of a mean of the distribution with those assumptions ) so the formulae were slightly different. But, more importantly, the concept is exactly the same. I wanted to understand it better myself so it was useful to me.

Oh, if you haven't already, I highly recommend either sitting in or maybe there's a two-semester mathematical-statistics class on the internet you could take. You sound interested and, when you learn it from a math-stat standpoint, there's nothing to memorize because it's more of a conceptual type of thing. In fact, if you take a good math-stat class with a good teacher and good book, the desire for memorization kind of gets lost in the shuffle.

$\endgroup$
43
  • $\begingroup$ > They are not because the ML procedure uses $n$ as the divisor when calculating $\tilde{\sigma}^2$. The OLS procedure on the other hand uses $(n-2)$. Therefore, the t-statistics generated by the two procedures will not be the same. < When using the ML procedure to estimate the standard deviation as input for the t-statistic, then one applies a correction to get an unbiased estimate. There is no rule that states that the t-statistic needs to be computed with the biased standard deviation. $\endgroup$ Jul 25, 2023 at 10:51
  • $\begingroup$ Are there examples of people that use a biased t-statistic and don't scale the statistic such that it is not t-distributed? $\endgroup$ Jul 25, 2023 at 10:58
  • $\begingroup$ mlofton: Thank you for coming back! Also, thank you for providing your version of answer. Okay, let's get to the business. It seems you mixed up $\tilde{\sigma}^2$ and $\hat{\sigma}^2$. But, let me just use your notation system, to make the discussion easier. That is, you use $\tilde{\sigma}^2$ to denote the estimated $\sigma^2$. Okay, I agree that $\tilde{\sigma}^2_{OLS} =\frac{\sum_{i=1}^n (y_i - \hat{\beta}_0-\hat{\beta}_1 x_i)^2}{n-2}$. And you construct $V_{OLS}=\frac{(n-2)\tilde{\sigma}^2_{OLS}}{\sigma^2} \sim \chi^2(n-2)$. That is correct, and I agree. (To be continued - Part 1) $\endgroup$
    – Will
    Jul 25, 2023 at 12:05
  • $\begingroup$ We know that $\tilde{\sigma}^2_{MLE} =\frac{\sum_{i=1}^n (y_i - \hat{\beta}_0-\hat{\beta}_1 x_i)^2}{n}$. To construct MLE version of $\chi^2$, you need to write it as $V_{MLE}=\frac{(n)\tilde{\sigma}^2_{MLE}}{\sigma^2} \sim \chi^2(n-2)$. (Part 2) $\endgroup$
    – Will
    Jul 25, 2023 at 12:09
  • $\begingroup$ Note that, we used $(n-2)$ in $V_{OLS}$ because $(n-2)$ in the denominator of $\tilde{\sigma}^2_{OLS} $. In a similar vein, we need to use $(n)$ in $V_{MLE}$ because $(n)$ in the denominator of $\tilde{\sigma}^2_{MLE}$ (Part 3) $\endgroup$
    – Will
    Jul 25, 2023 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.