# How can Logistic Regression produce curves that aren't traditional functions?

I think I have some fundamental confusion about how the functions in Logistic regression work (or maybe just functions as a whole).

How is it that the function h(x) produces the curve seen in the left of the image?

I see that this is a plot of two variables but then these two variables (x1 & x2) are also arguments of the function itself. I know standard functions of one variable map to one output but this function clearly isn't doing that-- and I'm not totally sure why.

My intuition is that the blue/pink curve isn't really plotted on this graph but rather is a representation (circles and X's) that get mapped to values in the next dimension (3rd) of the graph. Is this reasoning faulty and am I just missing something? Thanks for any insight/intuition.

• Pay attention to the axis labels, notice that neither is labeled $y$. Commented Jul 14, 2017 at 2:06
• What would a "traditional function" be?
– whuber
Commented Jul 14, 2017 at 14:13
• @matthewDrury I understand that, and this explains the 2D X/Os. I'm asking where plotted curve is then coming from Commented Jul 14, 2017 at 15:15

This is an example of overfitting on the Coursera course on ML by Andrew Ng in the case of a classification model with two features $(x_1, x_2)$, in which the true values are symbolized by $\color{red}{\large \times}$ and $\color{blue}{\large\circ},$ and the decision boundary is precisely tailored to the training set through the use of high order polynomial terms.

The problem that it tries to illustrate relates to the fact that, although the boundary decision line (curvilinear line in blue) doesn't mis-classify any examples, its ability to generalize out of the training set will be compromised. Andrew Ng goes on to explain that regularization can mitigate this effect, and draws the magenta curve as a decision boundary less tight to the training set, and more likely to generalize.

With regards to your specific question:

My intuition is that the blue/pink curve isn't really plotted on this graph but rather is a representation (circles and X's) that get mapped to values in the next dimension (3rd) of the graph.

There is no height (third dimension): there are two categories, $(\large\times$ and $\large\circ),$ and the decision line shows how the model is separating them. In the simpler model

$$h_\theta(x)=g\left(\theta_0 + \theta_1 \, x_1 + \theta_2 \, x_2 \right)$$

the decision boundary will be linear.

Perhaps you have in mind something like this, for example:

$$5 + 2 x - 1.3 x^2 -1.2 x^2 y + 1 x^2 y^2 + 3 x^2 y^3$$

However, notice that there is a $g(\cdot)$ function in the hypothesis - the logistic activation in your initial question. So for every value of $x_1$ and $x_2$ the polynomial function undergoes and "activation" (often non-linear, such in the a sigmoid function as in the OP, although not necessarily (e.g. RELU)). As a bounded output the sigmoid activation lends itself to a probabilistic interpretation: the idea in a classification model is that at a given threshold the output will be labeled $\large \times$ $\large($ or $\large \circ).$ Effectively, a continuous output will be squashed into a binary $(1,0)$ output.

Depending on the weights (or parameters) and the activation function, each point $(x_1,x_2)$ in the feature plane will be mapped to either the category $\large \times$ or $\large \circ$. This labeling may or may not be correct: they will be correct when the points in the sample drawn by $\color{red}{\large \times}$ and $\color{blue}{\large \circ}$ on the plane in the picture on the OP correspond to the predicted labels. The boundaries between regions of the plane labeled $\large \times$ and those adjacent regions labeled $\large \circ$. They can be a line, or multiple lines isolating "islands" (see by yourself playing with this app by Tony Fischetti part of this blog entry on R-bloggers).

Notice the entry in Wikipedia on decision boundary:

In a statistical-classification problem with two classes, a decision boundary or decision surface is a hypersurface that partitions the underlying vector space into two sets, one for each class. The classifier will classify all the points on one side of the decision boundary as belonging to one class and all those on the other side as belonging to the other class. A decision boundary is the region of a problem space in which the output label of a classifier is ambiguous.

There is no need for a height component to graph the actual boundary. If, on the other hand, you are plotting the sigmoid activation value (continuous with range $∈[0,1]),$ then you do need a third ("height") component to visualize the graph:

If you want to introduce a $3$D visualization for the decision surface, check this slide on an online course on NN's by Hugo Larochelle, representing the activation of a neuron:

where $y_1 = h_\theta(x)$, and $\mathbf W$ is the weight vector $(\Theta)$ in the example in the OP. Most interesting is the fact that $\Theta$ is orthogonal to the separating "ridge" in the classifier: effectively, if the ridge is a (hyper-)plane, the vector of weights or parameters is the normal vector.

Joining multiple neurons, these separating hyperplanes can be added and subtracted to end up with capricious shapes:

This links to the universal approximation theorem.

• +1 always enjoy reading your answer. It might be even better if you can have a decision plane intersect with your plot. to show some above and some under. Commented Jul 14, 2017 at 2:34
• Thanks so much for this. I still feel as though I'm missing something small about the curve itself-- is this to say that the decision boundary isn't really being "drawn" but rather is just Andrew Ng's way of indicating the value thresholds of x1 & x2 that cause the hypothesis to either be × or ∘? I think some of my confusion stemmed from how that curve could be a function in the first place, but I now realize it's not. Commented Jul 14, 2017 at 3:45
• @AntoniParellada This is great, I see the distinction now. Thanks a lot for the help. Commented Jul 14, 2017 at 15:38

We've got some heavy-duty mathematicians answering this question. I've never seen a diagram like you depict here, with the values for predictors X1 and X2 and the 'decision boundary' line separating the predicted positives from the predicted negatives. (or is it a map of predicted vs. actual outcomes?) But it is useful---as long as you have only two predictors of interest that you want to map.
It appears that the magenta line separates the predicted positives from the predicted negatives, whereas the dark blue line includes all the positives. This is usually the case in logistic regression: the model will correctly predict the outcome for less than 100% of the cases (and will predict some false positives and/or false negatives).
It's possible to run logistic regression and have the procedure produce the function h(x) for each individual case in the dataset. This will produce a propensity score for each subject, from 0 to 1, that gives the predicted likelihood or probability of the positive outcome for each subject based on that subject's predictor variables, based on the logistic regression model using all subjects. Those at the propensity score cutoff of 0.5 or above are predicted to have the outcome, and those below 0.5 are predicted to not have the outcome. But you can adjust this cutoff level as you see fit, for example to make a diagnostic prediction model of some outcome based on all the input variables that are entered in your logistic regression analysis. You can set the cutoff at 0.3 for example. You can then do a 2X2 table of predicted-vs-actual outcomes, and determine your sensitivity, specificity, false positive rate and false negative rate of the model based on this cutoff level. This provides more information and also frees you from the limit of 2 variables used in your graph. You can use as many predictors as you can reasonably fit in the model and still make a 2X2 table of actual-vs-predicted outcome. Since logistic regression uses categorical (yes-no) outcomes, each cell in the 2X2 table is simply a count of the subjects that meet the row and column criteria.
In the graph you provide, it probably assumes a cutoff of 0.5. This is the common default for software. If you adjusted it higher (to 0.65 for example), it might include all the O's inside the line but you would also have some false positives (X's that it thinks should be O's) that would be predicted by the model to have the outcome of interest. (or adjust the cutoff score lower and have more false negatives).
I hope this helps.