What does it mean to "fit" something using least squares? Let's say that I have the linear combinations
$$Z_m = \sum_{j = 1}^p \phi_{jm} X_j$$
for some constants $\phi_{1m}, \phi_{2m}, \dots, \phi_{pm}$, $m = 1, \dots, M$. It is then said that we can "fit" the linear regression model
$$y_i = \theta_0 + \sum_{m = 1}^M \theta_m z_{im} + \epsilon_i, \ \ \ i = 1, \dots, n,$$
using least squares.
Although I am not experienced in statistics, I do know what regression and least squares are (from a mathematical perspective). However, I'm struggling to reconcile my knowledge with what it means to "fit" the linear regression model using least squares, as described above. What does it mean to "fit" something using least squares, as in the above example?
 A: To “fit the model using least squares” means that we assume squared loss
$$
\mathcal{L}(\theta) = \sum_i ( y_i  - (\theta_0 + \sum_{m = 1}^M \theta_m z_{im}) )^2
$$
and we seek for minimum of it to find the estimates of the parameters
$$
\operatorname{arg\,min}_\theta \mathcal{L}(\theta)
$$
With linear regression, this can be done by applying linear algebra by ordinary least squares, while for other models you would use some optimization algorithm to find the minimum.
“Fitting“ means “finding the parameters”. It’s “fitting” because we seek for such parameters that make the model “fit” the data best.
A: Actually, assuming one is performing, at least, in part an exploratory model exercise, like involving specifying the best model form, possible explanatory variables, data transforms on those variables (to address normality and homogeneity of error variance, etc.) and even # of variables, a 'fitting' exercise, commonly occurs in stepwise least-squares context, for example. The latter process itself can be described in my, and the opinions of others, as a bit of an art, as opposed to pure science. In a reference article below employing the term empirical (whose definition includes "relying on experience or observation alone often without due regard for system and theory"), to quote:

Stepwise methods are quite common to be reported in empirically based journal articles..

So, in my opinion, the connotation of the word 'fitting', as in a stepwise method relating to empirical-based work, is applicable, at times. There are, however, valid special cases, where one is just updating coefficients in a previously postulated and tested model. Also, in more recent times with access to algorithms to enable advanced numerical and statistical analysis (like PCA based regression analysis, factor regression, and this discussed in a 2016 article Stepwise Regression and All Possible Subsets Regression in Education) developing good models with good explanatory variables is less of an art and more of a science.
I would also add in my opinion, however, that a knowledge background for the important aspect of specifying the best model form itself (not just relying on examination of goodness-of-fit metrics) is recommended to serve as a sound foundation in the appropriate context of physics, physical chemistry,..., as to why a proposed model form is, indeed, conceptually appropriate.
