1
$\begingroup$

I get the intuition behind the loss function for linear regression, which happens to be the MSE function. We have a set of points and we try to fit a line between them so that the line is at a minimum distance from all the points. So it makes sense to add up the distances from all the points and then try to minimize that distance.

However, in logistic regression, why do we take the sum of the loss function? Can't we calculate the losses for each training example, minimize that and move on to the next? We could then average out the weights for each example over all the iterations.

My question is: What is the intuitive sense behind taking the sum of the training example losses and them summing it up and minimizing that. It makes more sense to me to minimize the individual training example and then move on to the next.

$\endgroup$
1
  • 2
    $\begingroup$ Let's assume you have 100 training examples and 5 independent variables. If you look at each training example separately, you only need one independent variable, e.g., the constant term, with value $+/- \infty$ to predict that training example perfectly. The value of the constant term will change from training example to training example, so it is not in fact a constant. When you are done, you will have learned... nothing at all about the relationship between the independent variables and the target variable, and you will have a "model" with no predictive power. $\endgroup$
    – jbowman
    Apr 9 '18 at 16:27
1
$\begingroup$

Let's assume you have 100 training examples and 5 independent variables. If you look at each training example separately, you only need one independent variable, e.g., the constant term, with value $\pm \infty$ to predict that training example perfectly ($+\infty$ if the target value is $1$, $-\infty$ if it's $0$.) The value of the constant term will change from training example to training example, so it is not in fact a constant - but, because you are looking at the data one observation at a time, you have no way to force it to be a constant. When you are done, you will have learned... nothing at all about the relationship between the independent variables and the target variable, and you will have a "model" with 100% "explanatory" but no predictive power. You will have missed seeing the forest because you will have been looking at the trees, so to speak.

$\endgroup$

This site is temporarily in read only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .