I'm in an AP statistics class, and several weeks ago we discussed regression lines. I understand that a regression line is the best fit for the the given data and that it can be used to predict variables. A least squares regression line can be used to determine the smallest possible value for squared residuals. Why is a line informative even when the data doesn't fall along the line?
The objective of the least squares principle is to determine the parameters of the model such that the sum of the squared residuals is minimum. By residual we mean, the difference between the observed and the predicted value of the response variable. So, smallest squared residuals means the best possible prediction of the response variable. The regression model describes the existing functional relationship between the variables in the given set of observations.
Fitting a regression line to a given set of values means determining the parameter estimates of the model. These estimates are obtained to achieve some criterion. The criterion used in least squares principle is to obtain the parameter estimates such that the sum of the squared residuals is minimum. The regression line so obtained is the
best fit in the sense of minimum sum of squared residuals.
Apart from describing the relationship between the variables, another use of the regression model is to
predict the values of the response variable for some given values of the predictor variable.
It shed some light on unknown relationships in our life.
Suppose you are physics in 1600. You don't know what is free fall in Newtonian mechanics.
However, you are smart enough to figure out the falling distance $h$ has something to do with time $t$. But how?
You test many free fall at different heights, and get your data. You plot to see what's the relationship. It seems a linear model doesn't work.
$$h = \beta t$$
Then you tried
$$h = \beta t^2$$
This model matches your data much better. Then what is this $\beta$?
This $\beta$ can be guessed from your data. The best guess would the $\beta$ minimize the "distance" (forgive me) between the line and the data.
You find out $\beta = 5$ is the best answer. Then you will ask why 5? This is a magical number. Where does it come from? You begin your journey on finding reasonable explanation for this number. From now on, the question is no longer a statistics question. It is a physics question.
Our ultimate goal is to know what's behind what we see. However, science discovery is a long long journey. The fitted line could be a good starting point for us.