# What does it mean to "fit" something using least squares?

Let's say that I have the linear combinations

$$Z_m = \sum_{j = 1}^p \phi_{jm} X_j$$

for some constants $$\phi_{1m}, \phi_{2m}, \dots, \phi_{pm}$$, $$m = 1, \dots, M$$. It is then said that we can "fit" the linear regression model

$$y_i = \theta_0 + \sum_{m = 1}^M \theta_m z_{im} + \epsilon_i, \ \ \ i = 1, \dots, n,$$

using least squares.

Although I am not experienced in statistics, I do know what regression and least squares are (from a mathematical perspective). However, I'm struggling to reconcile my knowledge with what it means to "fit" the linear regression model using least squares, as described above. What does it mean to "fit" something using least squares, as in the above example?

• Informally, it just means that, given some data, we use a least squares algorithm to obtain estimates for the $\phi$. In general, when we say "fit" in the context of any statistical model, it means we just use some algorithm or procedure (in this case least squares) to estimate one or more coefficients. I will let the rigour police provide a more detailed answer :) Aug 27 '20 at 18:14
• @RobertLong Ahh, ok; I think that algorithmic perspective is what I am missing in my conceptualisation. I only understand this from a mathematical perspective, so it isn't clear to me what they meant by "fitting" something "using least squares"; but if we consider that it is done using an algorithm (numerical methods), then that statement starts to make more sense! Aug 27 '20 at 18:17
• What is your intended use of your current specified model or, are you first interested in getting a good model from the list of your variables pre-applications? Interested in producing forecasts, or just explaining historical oscillations? Aug 28 '20 at 20:24
• @AJKOER This is just a question about theory. I am not trying to apply anything – just learning. Aug 28 '20 at 20:25

To “fit the model using least squares” means that we assume squared loss

$$\mathcal{L}(\theta) = \sum_i ( y_i - (\theta_0 + \sum_{m = 1}^M \theta_m z_{im}) )^2$$

and we seek for minimum of it to find the estimates of the parameters

$$\operatorname{arg\,min}_\theta \mathcal{L}(\theta)$$

With linear regression, this can be done by applying linear algebra by ordinary least squares, while for other models you would use some optimization algorithm to find the minimum.

“Fitting“ means “finding the parameters”. It’s “fitting” because we seek for such parameters that make the model “fit” the data best.

• Tim: Welcome to the world of real applied statistics. Start by reading my answer! Aug 28 '20 at 17:56
• @AJKOER You use "fit" in an unusual sense. I believe most statisticians would describe your answer as a modeling exercise and would understand "fit" in the sense described here by Tim. Virtually nobody would understand "fit" to mean stepwise regression!
– whuber
Aug 28 '20 at 18:15
• @AJKOER you are confusing training/fitting model with model selection. As a side-note, using stepwise selection is not the best method for model selection either.
– Tim
Aug 28 '20 at 19:16
• @AJKOER no; “fitting model using least squares” does not mean model selection. It means that you take a particular model and seek for best set of parameters, not that you compare different models.
– Tim
Aug 28 '20 at 19:52
• @AJKOER ok, but your comments seem to imply that the things you describe mean “fitting the model” (the question), while they describe other procedures related to building models, what makes them misleading in terms of the question.
– Tim
Aug 29 '20 at 6:08

Actually, assuming one is performing, at least, in part an exploratory model exercise, like involving specifying the best model form, possible explanatory variables, data transforms on those variables (to address normality and homogeneity of error variance, etc.) and even # of variables, a 'fitting' exercise, commonly occurs in stepwise least-squares context, for example. The latter process itself can be described in my, and the opinions of others, as a bit of an art, as opposed to pure science. In a reference article below employing the term empirical (whose definition includes "relying on experience or observation alone often without due regard for system and theory"), to quote:

Stepwise methods are quite common to be reported in empirically based journal articles..

So, in my opinion, the connotation of the word 'fitting', as in a stepwise method relating to empirical-based work, is applicable, at times. There are, however, valid special cases, where one is just updating coefficients in a previously postulated and tested model. Also, in more recent times with access to algorithms to enable advanced numerical and statistical analysis (like PCA based regression analysis, factor regression, and this discussed in a 2016 article Stepwise Regression and All Possible Subsets Regression in Education) developing good models with good explanatory variables is less of an art and more of a science.

I would also add in my opinion, however, that a knowledge background for the important aspect of specifying the best model form itself (not just relying on examination of goodness-of-fit metrics) is recommended to serve as a sound foundation in the appropriate context of physics, physical chemistry,..., as to why a proposed model form is, indeed, conceptually appropriate.

• I am very thankful for the unexplained/unsourced downgrades for my qualifications on 'fitting' as I do want to set myself apart from others on this forum. Aug 28 '20 at 18:28
• I believe you can find hint for the votes in whuber comment to your comment.
– Tim
Aug 28 '20 at 19:18