This is a very basic question but I was reading the notes by Andrew NG (https://see.stanford.edu/materials/aimlcs229/cs229-notes1.pdf) about linear regression and am wondering how the initial guess for theta is made by algorithms such LinearRegression in python sklearn library or in general if I try to manually calculate this.

I understand using gradient descent will eventually lead to almost the same result for theta regardless of the initial guess but what I want to know is how these algorithms make the initial guess, be it for the intercept or the coefficient of one of the parameters.

(Here, theta refers to the coefficient for various predictor variables or the intercept)

  • 5
    $\begingroup$ Not sure about sklearn specifically, but most implementations of OLS regression do not use gradient descent like in those notes. This is because OLS has an analytical solution, and moreover, objects computed as part of that analytical solution have important statistical properties, so it makes sense to calculate those anyways. The purpose of those notes is to demonstrate a general principle, not to claim that this is how you should actually fit a linear regression model. $\endgroup$ – stats_model Mar 24 at 14:24
  • 1
    $\begingroup$ sklearn's vanilla LinearRegression uses LAPACK, and so (if I understand correctly) does not use gradient descent. However, their penalized versions, as well as logistic regression (penalized or not) do use either gradient descent or coordinate descent. $\endgroup$ – Ben Reiniger Mar 24 at 14:34
  • $\begingroup$ @stats_model could you please elaborate on what you mean by an analytical solution for OLS? I'm curious to know the other methods used to fit a linear regression model. $\endgroup$ – user42 Mar 24 at 15:13
  • 1
    $\begingroup$ user42, see stats.stackexchange.com/search?q=regression+normal+equations for the standard formulas. Generally, when an analytical solution does not exist (as in the penalized versions referred to by @Ben), one can construct an initial estimate from a simpler version of the problem, such as setting the penalty to zero and using the analytical solution, using a "warm start" based on the previous result (when varying the penalty), or by solving the problem with a subset of the data. $\endgroup$ – whuber Mar 24 at 16:18

This might depend on the solver, but the two places I see sklearn explicitly initializing coefficients, it sets them to zero:

if not self.warm_start or not hasattr(self, "coef_"):
    coef_ = np.zeros((n_targets, n_features), dtype=X.dtype,

source for the above, in ElasticNet.fit, and similar source in enet_path, the coordinate descent elastic-net path function.

Their logistic regression models use external solvers like liblinear, saga, etc., and I'm not sure how those are initialized. the LIBLINEAR paper only mentions initialization as "Given initial $w\in R^n$."

Setting your own initial guess can be accomplished when the sklearn estimator has a warm_start option, as you might guess from the above code snippet: just manually set the attribute coef_ to your initial guess.

  • $\begingroup$ Aha, could it be that they use the values of the null hypothesis as initial values for the coefficients? $\endgroup$ – user42 Mar 24 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.