1
$\begingroup$

For simplicity, let's assume that we have the true DGP

$$y_i= \beta_0 + \beta_1x_i + \epsilon_i$$ where $\epsilon_i \sim N(0, \sigma^2)$ $(i = 1,2,...,n)$

Assume that these following usual assumptions meet (for the classical linear regression model)

  1. $y_i= \beta_0 + \beta_1x_i + \epsilon_i$ (linearity)
  2. $x_i$ is deterministic
  3. $\epsilon_i \sim N(0, \sigma^2)$ and $Cov[\epsilon_i, \epsilon_j] = 0$ for $i \neq j$ (normality, homoskedasticity and uncorrelatedness)

If we use Maximum Likelihood Estimation Method (MLE) to estimate $\beta_0$ and $\beta_1$, is it correct to state that the nice asymptotic properties as n approaches infinity (asymptotic efficiency, asymptotic consistency, ...) still hold for the MLE estimators of $\beta_0$ and $\beta_1$?

I believe that the answer is no because of the second assumption and, for example, $MSE[\hat{\beta}_{1MLE/OLS}] = \mathbb{V}[\hat{\beta}_{1MLE/OLS}] = \frac{\sigma^2}{Dev(x)}$ which implies there is no guarantee that $MSE[\hat{\beta}_{1MLE/OLS}]$ will approach 0 as n approaches infinity (sufficient but not necessary condition for convergence in probability), but I want to know whether or not it is correct.

$\endgroup$
6
  • $\begingroup$ What makes you think that with simple linear regression, the MLE estimators are any different from minimising the sum of squares of the residuals? $\endgroup$
    – Henry
    Commented Dec 2, 2022 at 0:42
  • $\begingroup$ @Henry I want to know about the asymptotic properties of the estimators when n goes to infinity. Since the regressor is determinstic. I think there is no guarantee that MSE of the estimators, for example for MLE estimator of $\beta_1$, will go to 0 (sufficient but not necessary condition for convergence in probability - or here, consistency) $\endgroup$
    – Neuchâtel
    Commented Dec 2, 2022 at 0:46
  • $\begingroup$ @edelweiss (Unless I've missed something, ) since the estimators depend on $\epsilon$, they are not deterministic even if $x$ is set manualy, for example in an experimental setting. Furthermore, there is no need for asymptotics in your setting: we get joint normality with the standard moments for all finite samples as well as optimality even in more general settings by the Gauss Markov theorem (also in finite samples). $\endgroup$ Commented Dec 2, 2022 at 0:52
  • $\begingroup$ Is this an example of non-MSE consistent but consistent (convergence in probability ) estimators? @JohnMadden. Anyway, thank you so much! $\endgroup$
    – Neuchâtel
    Commented Dec 2, 2022 at 1:02
  • $\begingroup$ It depends on what additional $x_i$ you include as $n$ increases. The variance of $\hat \beta_1$ is $\frac{\sigma^2}{\sum(x_i-\bar x)^2}$ which tends towards $0$ so long as the denominator increases without bound and you do not concentrate the additional $x_i$ around the existing $\bar x$. So you are correct to worry if $x_1=1$ and $x_2=-1$ and all the other $x_i=0$: you will never mitigate the impact of $\epsilon_1$ and $\epsilon_2$. But this is not realistic: the point of expanding the sample for OLS is to see the impact of different values of $x_i$ on $y_i$. $\endgroup$
    – Henry
    Commented Dec 2, 2022 at 1:10

1 Answer 1

1
$\begingroup$

Requested in comments

With simple linear regression, the MLE estimators the same as those from minimising the sum of squares of the residuals.

It depends on what additional $x_i$ you include as $n$ increases. The variance of $\hat \beta_1$ is $\frac{\sigma^2}{\sum(x_i-\bar x)^2}$ which tends towards $0$ so long as the denominator increases without bound and you do not concentrate the additional $x_i$ around the existing $\bar x$.

So for example you are correct to worry if $x_1=1$ and $x_2=-1$ and all the other $x_i=0$: you will never mitigate the impact of $\epsilon_1$ and $\epsilon_2$ on $\hat \beta_1$.

But this is not realistic: the point of expanding the sample for OLS is to see the impact of different values of $x_i$ on $y_i$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.