What is the purpose of giving a probabilistic interpretation of linear and logistic regression?

Question

I can write the cost functions for both linear regression and logistic regression and minimize them using gradient descent without using any probabilistic notions. In the case of linear regression the cost function is:

$$ \displaystyle J(\theta) = \frac{1}{2m} \displaystyle\sum\limits_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})^2 $$

For logistic regression the cost function is: $$ \displaystyle J(\theta) = -\frac{1}{m} \left[\sum\limits_{i=1}^m y^{(i)}log(h_\theta(x^{(i)})+(1-y^{(i)})log(1-h_\theta(x^{(i)})) \right] $$

The intuition for both cost functions is to penalize models which have large errors between the real response and the predicted response. One other important property of these functions is that they are convex which makes it easy to find the global minimums.

These cost functions can also be derived using the maximum likelihood method under certain assumptions. But what is the purpose of giving them a probabilistic interpretation? Are there any (theoretical or practical) benefits?

Answer 1

Short answer:

• If we have a large testing data set (say 1 million samples), "probabilistic interpretation" does not bring us lots of value. Because the performance on large testing data tells everything.

• If we have a small testing data set (say 1000 samples), probabilistic interpretation tell us how reliable the model is. In other words: what's the chance of the model and estimated coefficients are significant. Or we are just capturing some noise in the data.

---

Long answer:

From your notation, I guess you learned the linear regression and logistic regression from machine learning (and may be from Coursera Andrew NG's course) but not statistics.

If you learned these from Coursera, what you learned is really a simplified version of linear method. It emphasize a lot on optimization. Andrew NG is teaching must to known and very practical tricks to let people to learn it faster, without too much details, and can apply it in real world problem.

In fact, linear regression and logistic regression are invented much easier before the machine learning era. Least squares can go back for two hundred years. In that time, computers even do not exist. Statisticians work with pencil and papers to develop these methods. In addition, during that time, we do not have too many data. The famous IRIS data in 1936 paper only has 150 data points.

Usually, less data means more math: it is easy to calculate the coefficient, but so what? people also want to say "how good" are my coefficients in absence of large testing data. Therefore, there are many assumptions and very nice properties on linear regression and logistic regression. All of these nice properties coming form "probabilistic interpretation".

Historically, when people use these methods, people need to carefully check the assumptions to make sure it works properly. Which is the mindset of the statisticians. With huge data today, many people try to do the shortcut way. In stead of derive properties from math, trying it to use it with large /reliable testing data. The idea is if the testing data is huge, I do not need to "estimate" how reliable my model is.

we can downplay to derive the math properties, such as confidence interval, p value, etc. Because it is easy to check if the model works in real world without too much math.

But with less data, it is very important to get a sense of how good we are statistically given the estimated parameters.