Linear regression (specifically, least squares) is usually motivated as a linear approximation of the conditional expectation function (CEF), and can be shown to be the minimum mean square error linear approximation of the CEF. Was this the motivating reason for deriving the least squares estimator, or was it just an intuitive way to fit a line through the cloud of data, and later on it was proven to be a 'good' approximation of the CEF?

  • 2
    $\begingroup$ This language (especially the acronym CEF) are so far as I know common only in econometrics, and so the "usually" here is dubious. The history is even in summary very messy. Least squares (Gauss, Legendre, etc.) and regression (Galton, etc.) have largely separate origins at different points in the 19th century and it took some time for it to be clear that they are (or more precisely can be) regarded as intimately related. Galton so far as I know never used least squares, for example. Fitting straight lines is a yet older idea. The history is very well covered in S.M. Stigler's 1986 book. $\endgroup$
    – Nick Cox
    Nov 21, 2020 at 11:24
  • 2
    $\begingroup$ The full title of the book is Stigler "The History of Statistics: The Measurement of Uncertainty before 1900" (1986). $\endgroup$ Nov 21, 2020 at 12:22
  • 1
    $\begingroup$ A positive reply is that Galton's intent -- and Galton wasn't first here but he did use the term "regression" -- was close to finding response given outcome starting from a scatter plot of bivariate data. In that sense he was closer to some modern thinking than were (in particular) those of his successors who wanted to firm up the problem as fitting a bivariate normal. Around 1900, Yule was much closer than Karl Pearson to mainstream modern thinking. $\endgroup$
    – Nick Cox
    Nov 21, 2020 at 13:01


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.