I can write the cost functions for both linear regression and logistic regression and minimize them using gradient descent without using any probabilistic notions. In the case of linear regression the cost function is:

$$ \displaystyle J(\theta) = \frac{1}{2m} \displaystyle\sum\limits_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})^2 $$

For logistic regression the cost function is: $$ \displaystyle J(\theta) = -\frac{1}{m} \left[\sum\limits_{i=1}^m y^{(i)}log(h_\theta(x^{(i)})+(1-y^{(i)})log(1-h_\theta(x^{(i)})) \right] $$

The intuition for both cost functions is to penalize models which have large errors between the real response and the predicted response. One other important property of these functions is that they are convex which makes it easy to find the global minimums.

These cost functions can also be derived using the maximum likelihood method under certain assumptions. But what is the purpose of giving them a probabilistic interpretation? Are there any (theoretical or practical) benefits?

  • 2
    $\begingroup$ What is the purpose of a 3 legged stool? It is to sit on it. Utility! $\endgroup$
    – Chris
    Nov 20, 2017 at 12:59
  • $\begingroup$ What are you using your regression and logistic regression models to do? What are you using them for? $\endgroup$
    – Glen_b
    Nov 20, 2017 at 14:21
  • $\begingroup$ @Glen_b I do not have a specific application but I will generally use them in machine learning tasks for prediction $\endgroup$
    – Sanyo Mn
    Nov 20, 2017 at 14:41
  • 2
    $\begingroup$ Are you asking why predicting probabilities is the correct thing to do, or why viewing the models as fitting conditional distributions is useful? These are both important questions, but I'm not totally clear which one you are getting at. $\endgroup$ Nov 20, 2017 at 16:39
  • 1
    $\begingroup$ @MatthewDrury, my question is closer to the second one. $\endgroup$
    – Sanyo Mn
    Nov 20, 2017 at 18:18

1 Answer 1


Short answer:

  • If we have a large testing data set (say 1 million samples), "probabilistic interpretation" does not bring us lots of value. Because the performance on large testing data tells everything.

  • If we have a small testing data set (say 1000 samples), probabilistic interpretation tell us how reliable the model is. In other words: what's the chance of the model and estimated coefficients are significant. Or we are just capturing some noise in the data.

Long answer:

From your notation, I guess you learned the linear regression and logistic regression from machine learning (and may be from Coursera Andrew NG's course) but not statistics.

If you learned these from Coursera, what you learned is really a simplified version of linear method. It emphasize a lot on optimization. Andrew NG is teaching must to known and very practical tricks to let people to learn it faster, without too much details, and can apply it in real world problem.

In fact, linear regression and logistic regression are invented much easier before the machine learning era. Least squares can go back for two hundred years. In that time, computers even do not exist. Statisticians work with pencil and papers to develop these methods. In addition, during that time, we do not have too many data. The famous IRIS data in 1936 paper only has 150 data points.

Usually, less data means more math: it is easy to calculate the coefficient, but so what? people also want to say "how good" are my coefficients in absence of large testing data. Therefore, there are many assumptions and very nice properties on linear regression and logistic regression. All of these nice properties coming form "probabilistic interpretation".

Historically, when people use these methods, people need to carefully check the assumptions to make sure it works properly. Which is the mindset of the statisticians. With huge data today, many people try to do the shortcut way. In stead of derive properties from math, trying it to use it with large /reliable testing data. The idea is if the testing data is huge, I do not need to "estimate" how reliable my model is.

we can downplay to derive the math properties, such as confidence interval, p value, etc. Because it is easy to check if the model works in real world without too much math.

But with less data, it is very important to get a sense of how good we are statistically given the estimated parameters.

  • 2
    $\begingroup$ "If we have a large testing data set (say 1 million samples), "probabilistic interpretation" does not bring us lots of value. Because the performance on large testing data tells everything." This begs the question: what is performance? If it's purely judged on classification, than yes. But performance of such a model is measured many different ways than single-object classification. $\endgroup$
    – Paul
    Nov 20, 2017 at 15:49
  • $\begingroup$ Thanks for this insightful answer, I see that the amount of data is the key here, and yes I followed Andrew Ng's courses. $\endgroup$
    – Sanyo Mn
    Nov 20, 2017 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.