Parabolic boundary with logistic regression

Question

Is it possible to generate a parabolic boundary with logistic regression? If so, how? If not, why and how can we generate the most appropriate boundary?

Below a snippet of Python code that generates a dummy example of the data to classify.

x_1 = np.arange(0., 25. , 0.1)
x_2_partial_1 = x_1**2 + np.reshape(np.random.normal(10, 5, len(x_1)), len(x_1), 1)
x_2_partial_2 = x_1**2 + np.reshape(np.random.normal(10, 5, len(x_1)), len(x_1), 1) + 100

plt.figure(figsize = (20, 10))
plt.plot(x_1, x_2_partial_1, 'r+', x_1, x_2_partial_2, 'b_')
plt.show()

dataset_stg_1 = np.column_stack((x_1, x_2_partial_1, np.zeros(len(x_1))))
dataset_stg_2 = np.column_stack((x_1, x_2_partial_2, np.ones(len(x_1))))
dataset = np.concatenate((dataset_stg_1, dataset_stg_2), axis = 0)

Answer 1

Yes and no. Technically, logistic regression can only find a linear decision boundary, so the technical answer is no. However, you can achieve the same effect by mapping your data into a higher dimensional space where the decision boundary is linear, or put simply for this case, by including x_1**2 as a feature in your logistic regression. Some example code to demonstrate this:

Note, you should use a train test split, this code is just to illustrate the point:

from sklearn.linear_model import LogisticRegression

X = dataset[:,0:-1]
y = dataset[:,-1]

lm_1 = LogisticRegression()
lm_2 = LogisticRegression()

# Fitting on x_1 and x_2
lm_1.fit(X,y)
y_hat_1 = lm_1.predict(X)

# Fitting on x_1, x_2 AND x_1**2 (you could even leave out x_1 in this case)
lm_2.fit(np.column_stack((X,X[:,0]**2)),y)
y_hat_2 = lm_2.predict(np.column_stack((X,X[:,0]**2)))

fig, ax = plt.subplots(ncols=2,figsize = (20, 10))
ax[0].plot(X[y_hat_1==1,0], X[y_hat_1==1,1], 'r+', X[y_hat_1==0,0], X[y_hat_1==0,1], 'b_')
ax[1].plot(X[y_hat_2==1,0], X[y_hat_2==1,1], 'r+', X[y_hat_2==0,0], X[y_hat_2==0,1], 'b_')
plt.show()

So as you can see, by adding x_1 squared as a feature, the data become linearly separable. If you were to project this linear decision boundary from this higher dimensional space (well, maybe not higher dimensional, but a nonlinear mapping of your original space) back onto your original space, it becomes parabolic. We can demonstrate it by making predictions over a grid to illustrate the decision boundary (borrowing code from https://stackoverflow.com/a/28257799/1011724):

fig, ax = plt.subplots(ncols=2, figsize=(20, 10))

xx, yy = np.mgrid[0:26:0.01, 0:800:1]
grid = np.c_[xx.ravel(), yy.ravel()]
probs = lm_1.predict(grid).reshape(xx.shape)

contour = ax[0].contourf(xx, yy, probs, 25, cmap="Pastel1", vmin=0, vmax=1)
ax[0].plot(X[y_hat_1==1,0], X[y_hat_1==1,1], 'r+', X[y_hat_1==0,0], X[y_hat_1==0,1], 'b_')
ax[0].set(xlim=(0, 25), ylim=(0, 800), xlabel="$X_1$", ylabel="$X_2$")


xx, yy = np.mgrid[0:26:0.01, 0:800:1]
grid = np.c_[xx.ravel(), yy.ravel(), xx.ravel()**2]
probs = lm_2.predict(grid).reshape(xx.shape)

contour = ax[1].contourf(xx, yy, probs, 25, cmap="Pastel1", vmin=0, vmax=1)
ax[1].plot(X[y_hat_2==1,0], X[y_hat_2==1,1], 'r+', X[y_hat_2==0,0], X[y_hat_2==0,1], 'b_')
ax[1].set(xlim=(0, 25), ylim=(0, 800), xlabel="$X_1$", ylabel="$X_2$");

Bu tnote that while it appears to have found a parabolic decision boundary in your original feature space, it has actually found a linear decision boundary in a new feature space. A nice way to see this is to plot x_2 against x_1**2 where you can clearly see the decision boundary is linear:

x_1 = np.arange(0., 25. , 0.1)
x_2_partial_1 = x_1**2 + np.reshape(np.random.normal(10, 5, len(x_1)), len(x_1), 1)
x_2_partial_2 = x_1**2 + np.reshape(np.random.normal(10, 5, len(x_1)), len(x_1), 1) + 100

plt.figure(figsize = (20, 10))
plt.plot(x_1**2, x_2_partial_1, 'r+', x_1**2, x_2_partial_2, 'b_')
plt.show()