1
$\begingroup$

Is it possible to generate a parabolic boundary with logistic regression? If so, how? If not, why and how can we generate the most appropriate boundary?

Below a snippet of Python code that generates a dummy example of the data to classify.

x_1 = np.arange(0., 25. , 0.1)
x_2_partial_1 = x_1**2 + np.reshape(np.random.normal(10, 5, len(x_1)), len(x_1), 1)
x_2_partial_2 = x_1**2 + np.reshape(np.random.normal(10, 5, len(x_1)), len(x_1), 1) + 100

plt.figure(figsize = (20, 10))
plt.plot(x_1, x_2_partial_1, 'r+', x_1, x_2_partial_2, 'b_')
plt.show()

dataset_stg_1 = np.column_stack((x_1, x_2_partial_1, np.zeros(len(x_1))))
dataset_stg_2 = np.column_stack((x_1, x_2_partial_2, np.ones(len(x_1))))
dataset = np.concatenate((dataset_stg_1, dataset_stg_2), axis = 0)

enter image description here

$\endgroup$

1 Answer 1

2
$\begingroup$

Yes and no. Technically, logistic regression can only find a linear decision boundary, so the technical answer is no. However, you can achieve the same effect by mapping your data into a higher dimensional space where the decision boundary is linear, or put simply for this case, by including x_1**2 as a feature in your logistic regression. Some example code to demonstrate this:

Note, you should use a train test split, this code is just to illustrate the point:

from sklearn.linear_model import LogisticRegression

X = dataset[:,0:-1]
y = dataset[:,-1]

lm_1 = LogisticRegression()
lm_2 = LogisticRegression()

# Fitting on x_1 and x_2
lm_1.fit(X,y)
y_hat_1 = lm_1.predict(X)

# Fitting on x_1, x_2 AND x_1**2 (you could even leave out x_1 in this case)
lm_2.fit(np.column_stack((X,X[:,0]**2)),y)
y_hat_2 = lm_2.predict(np.column_stack((X,X[:,0]**2)))

fig, ax = plt.subplots(ncols=2,figsize = (20, 10))
ax[0].plot(X[y_hat_1==1,0], X[y_hat_1==1,1], 'r+', X[y_hat_1==0,0], X[y_hat_1==0,1], 'b_')
ax[1].plot(X[y_hat_2==1,0], X[y_hat_2==1,1], 'r+', X[y_hat_2==0,0], X[y_hat_2==0,1], 'b_')
plt.show()

enter image description here

So as you can see, by adding x_1 squared as a feature, the data become linearly separable. If you were to project this linear decision boundary from this higher dimensional space (well, maybe not higher dimensional, but a nonlinear mapping of your original space) back onto your original space, it becomes parabolic. We can demonstrate it by making predictions over a grid to illustrate the decision boundary (borrowing code from https://stackoverflow.com/a/28257799/1011724):

fig, ax = plt.subplots(ncols=2, figsize=(20, 10))

xx, yy = np.mgrid[0:26:0.01, 0:800:1]
grid = np.c_[xx.ravel(), yy.ravel()]
probs = lm_1.predict(grid).reshape(xx.shape)

contour = ax[0].contourf(xx, yy, probs, 25, cmap="Pastel1", vmin=0, vmax=1)
ax[0].plot(X[y_hat_1==1,0], X[y_hat_1==1,1], 'r+', X[y_hat_1==0,0], X[y_hat_1==0,1], 'b_')
ax[0].set(xlim=(0, 25), ylim=(0, 800), xlabel="$X_1$", ylabel="$X_2$")


xx, yy = np.mgrid[0:26:0.01, 0:800:1]
grid = np.c_[xx.ravel(), yy.ravel(), xx.ravel()**2]
probs = lm_2.predict(grid).reshape(xx.shape)

contour = ax[1].contourf(xx, yy, probs, 25, cmap="Pastel1", vmin=0, vmax=1)
ax[1].plot(X[y_hat_2==1,0], X[y_hat_2==1,1], 'r+', X[y_hat_2==0,0], X[y_hat_2==0,1], 'b_')
ax[1].set(xlim=(0, 25), ylim=(0, 800), xlabel="$X_1$", ylabel="$X_2$");

enter image description here

Bu tnote that while it appears to have found a parabolic decision boundary in your original feature space, it has actually found a linear decision boundary in a new feature space. A nice way to see this is to plot x_2 against x_1**2 where you can clearly see the decision boundary is linear:

x_1 = np.arange(0., 25. , 0.1)
x_2_partial_1 = x_1**2 + np.reshape(np.random.normal(10, 5, len(x_1)), len(x_1), 1)
x_2_partial_2 = x_1**2 + np.reshape(np.random.normal(10, 5, len(x_1)), len(x_1), 1) + 100

plt.figure(figsize = (20, 10))
plt.plot(x_1**2, x_2_partial_1, 'r+', x_1**2, x_2_partial_2, 'b_')
plt.show()

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.