0
$\begingroup$

In a simple linear regression, Deriving OLS estimators by maximum likelihood principle (shown in this link Derivation of OLS Estimator) takes the form

enter image description here

Does this mean that coefficient estimates are dependant on other coefficients; therefore chronology matters?

  • If no, how do I solve for b0 and b1 from the above equations?
  • If yes, which coefficient gets estimated first? (Is it the intercept first followed by predictor estimates? or other way round? Also within the 'predictor coefficients' does order matter?,If yes: which one gets estimated first?)

(I believe the matrix form i.e. normal equation used in OLS suggests that any coefficient estimate is dependant on 'other predictors' but not their 'beta values'. So chronology shouldn't matter)

$\endgroup$

2 Answers 2

4
$\begingroup$

What you have here is a set of two simultaneous equations with two unknowns. Assuming that the explanatory variable in the regression is not constant (so that the design matrix has linearly independent columns), these equations have unique explicit solutions. (The solution for the slope coefficient is given in the linked notes you gave.) The explicit forms for the coefficients are:

$$\begin{align} \hat{\beta}_0 &= \bar{y} - \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} \cdot \bar{x}, \\[10pt] \hat{\beta}_1 &= \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2}. \\[6pt] \end{align}$$

Since either coefficient can be computed directly from the underlying data, there is no particular need to compute one of these coefficients before the other. Since the intercept coefficient has a simple form in terms of the slope coefficient, it is usually simpler to compute the slope coefficient first, but this is not a necessity.

$\endgroup$
3
$\begingroup$

Coefficient estimates depend on which independent variables (i.e. the $x_i$'s) you include in your regression (not the coefficients explicitly). Which independent variables you include impacts how the 'terrain' of the cost function (i.e. the error sum of squares) looks, and hence where the minimum of the cost function occurs.

The parameters are estimated jointly. This means you estimate them all at the same time. When you estimate a regression equation with software, a gradient based method traverses the terrain of the cost function, updating the estimates all at the same time while it looks for the minimum. One coefficient isn't estimated before another. It all happens at the same time.

$\endgroup$
1
  • $\begingroup$ Least squares gets there in one step... $\endgroup$
    – Nick Cox
    Mar 22, 2020 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.