I am confused about masking in linear regression for multiple classes...At the bottom I have added one of the most used examples. There are two points I do not understand

(1) Why is on the image at the bottom only one "line" visible as decision boundary? I would have expected three lines - namely one for each regression model, as indicated by this picture:

My Interpretation of the regression model

Picture from the book The elements of statistical learning:

Picture is taken from https://towardsdatascience.com/classification-a-linear-approach-part-1-b080c13992dd

(2) In the book is also added a plot of the error function. But honestly, I don't understand it... I do not get why the error function of class green is horizontal. And I would have expected that the error functions for class blue and red have exactly the opposite slope. Error function

Thanks so much in advance for your help! I have read almost all blog posts I found regarding this topic, but somehow I do not get it...


1 Answer 1


I can't expand on what is already discussed in ESL, but hopefully, I can explain what the book is saying in a way that you can better understand. First, I'll clarify that the approach they are describing is using linear regression to predict the binary variable of whether a point belongs in a given class. The book fits a separate regression line for each class. The point is not to predict X2 from X1, which seems to be what you are attempting with the regression lines in your first image, but to predict the class, or the color of the point, given both X1 and X2. Here is an example of what the data and corresponding regression line for the first (red) class of points approximately look like.

Edit: Note, I am only plotting class versus a single X variable. The actual data is 2-Dimensional (has X1 and X2). I represent the regression as a line but it would actually be a plane. I believe this simplified example clarifies the idea nevertheless.

Regression of whether a point belongs to class one (red) vs X

The third image you include is not a graph of the error of the fits. It is actually a graph of the regression lines themselves. The rug plot (colored dashes on the X-axis) indicates at which values the points for each class are equal to 1. The graph I included is meant to be analogous to the red regression line in the left pane.

enter image description here

Because class one (red line) is equal to one at only low values of X and zero elsewhere, the slope is negative. Because class three (blue line) is equal to one at only high values of X and zero elsewhere, it has a positive slope. However, class 2 (green line) is equal to one only around the middle X's and is equal to zero for both low and high values. Therefore, the slope of the green regression line is very close to zero. Regression of whether a point belongs to class two (Green) vs X

To classify a point as belonging to a class, we consider the predicted value of the point using each of the three regression lines. If the red regression line predicts the highest value, then the point is assigned to class 1. In your third image, the three regression lines are plotted together and you can see that at every point, either the red or the blue regression line is higher than the green regression line. Therefore, we will never predict that a point is green.

You create three regression lines, but decision boundaries do not correspond 1:1 with the regression lines. They indicate the boundaries at which one regression line is greater than another so that moving from one side of the decision boundary to the other, you will go from predicting one class to another. The values of the regression predictions are used to create the decision boundary displayed in the left panel of your third image. Because class 2 is never predicted, you are able to separate the predictions with only a single line. To the left of the line, the red regression line is highest and so you predict class 1. To the right, you predict class 3.

  • $\begingroup$ Thank you so, so much for the extremely detailed answer! :-) I did not get it and now it is clear to me! Only to clarify: So the first picture from the book I added shows only the decision boundary, which is constructed on base of the three regression models which are shown in the second picture from the book? $\endgroup$
    – user290388
    Jul 4, 2020 at 15:57
  • 1
    $\begingroup$ I'm glad it helped. You are correct, but, like me, the book simplified the problem slightly in the bottom graph and only showed the regression lines in 1-dimension. If you drew a decision boundary on top of the bottom graph, it would just be a straight line down the middle. You can see that in the first graph from the book, the actual decision boundary is a diagonal line, which means that it is considering both X1 and X2. But generally, the predicted values from the three regression lines determine the decision boundary. $\endgroup$
    – Ryan Volpi
    Jul 4, 2020 at 16:05
  • $\begingroup$ Thank you! Your explanations helped me a lot! :) $\endgroup$
    – user290388
    Jul 5, 2020 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.