# Is there a "piecewise linear fitting" for logistic regression?

For regression problem we can fit the data with a piecewise linear function (Linear Splines). Is there a "piecewise linear fitting" for binary classification?

Is that using spline basis expansion for logistic regression?

For example, some model to fit data like this (for logistic regression the decision boundary is a line. in the picture, the decision boundary is a piecewise linear function.)

• What exactly is the image supposed to show? Do I assumed correctly that $x$ and $y$ are some real-valued features and you are predicting classes $z$ indicated by blue vs red colour? How does it relate to piecewise-linear model?
– Tim
Commented Jun 16, 2020 at 8:38
• so splines are added to linear/logistic regression etc by just providing extra inputs which are non linear transformations of the input (eg max(x-6,0) and max(y-.3,0) much as you would add eg quadratic terms. for your example I am guessing you would need interaction terms. but afaik, if you can generate the same decision boundary with linear regression (and a threshold) then you can just add the "logistic function on top of that" Commented Jun 16, 2020 at 21:12

Yes, what you're describing is a model where the predicted probability of the positive class is obtained by passing a piecewise linear function of the input through the logistic sigmoid function. That is:

$$p(y=1 \mid x) = \frac{1}{1 + \exp(-\phi(x))}$$

where $$y \in \{0,1\}$$ is the class label, $$x \in \mathcal{X}$$ is the input, and $$\phi: \mathcal{X} \to \mathbb{R}$$ is a piecewise linear function. Note that ordinary logistic regression is a special case, where $$\phi(x) = w \cdot x$$.

Neural nets with piecewise linear activation functions (e.g. ReLU, PReLU) and sigmoidal output units are a common form of this model. In this case, supposing $$h(x)$$ is a vector of activations in the last hidden layer, and $$w$$ and $$b$$ are the weights and bias of the output unit, then $$\phi(x) = w \cdot h(x) + b$$.

Gradient boosted decision trees are another common form. In this case, $$\phi(x) = \sum_{i=1}^k w_i f_i(x)$$ where each $$f_i(x)$$ is a decision tree with weight $$w_i$$. And, the trees and weights are learned sequentially by gradient boosting. Here, the piecewise linear components are usually parallel to the axes of the input space, because decision trees typically split along a single feature at a time. However, variants that split using oblique hyperplanes are also possible.

When using these models, we don't typically believe that the decision boundary is truly piecewise linear (as in your example). Rather, they're useful because piecewise linear functions can approximate arbitrary decision boundaries, while being fast to compute and efficient to learn.

I'm assuming that you have in mind that the number of "knots" (pieces of the piecewise linear function) are known, but their locations are not.

Here are two ideas.

## Decision trees

Vanilla decision trees (trivially) form piecewise (axis-aligned) decision boundaries, but I don't think that's what you had in mind.

"Multivariate Decision Trees" form piecewise linear decision boundaries, which I'd guess is more what you're looking for. (Figure 1 from that paper below)

• Solid line - decision boundary of vanilla decision tree
• dashed line - decision boundary of multivariate tree

## Neural nets

I know, I know, deep neural nets these days aren't usually interpretable, but very small, shallow architectures can be interpretable. If you have in mind that the knot locations are learnable, then I think it's a nice framework to work in.

Your example can be solved with the composition of two (sets) of logistic regressions ( an ANN, with one hidden layer having two neurons ) These two hidden layers implement these two decision boundaries. These have the effect of mapping your red points to the origin, and blue points to one of $$(0,1), (1,0),(1,1)$$.

The last "layer" just has to separate the origin from everything else and would not even need to be learned.

Edit: of course just because a network can learn this, doesn't mean it will.