Different results for Logistic Regression (wrong) and Perceptron (correct)

Question

To help me with some understanding, I'm trying to learn the Logical AND and Logical OR using Linear Regression trained over the following data:

import numpy as np
from sklearn.linear_model import LogisticRegression, Perceptron

X = np.array([
    [0, 0],
    [0, 1],
    [1, 0],
    [1, 1]
])

y_or = np.array([0, 1, 1, 1])
y_and = np.array([0, 0, 0, 1])

Intuitively, I don't see any problem as the data points are linearly separable w.r.t. to their class 0 or 1. For comparison, I also tried the Perceptron algorithm, which returns the correct predictions. However, the Linear Regression gets it wrong

# Logistic Regression
clf = LogisticRegression().fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 0)

clf = LogisticRegression().fit(X, y_or)
clf.predict(X) # --> array([1, 1, 1, 1)


# Perceptron
clf = Perceptron().fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 1])

clf = Perceptron().fit(X, y_or)
clf.predict(X) # --> array([0, 1, 1, 1])

I feel I'm missing something silly here. Or my understanding is simply way off. I would expect that Linear Regression and Perceptron have similar decision boundaries -- of course, the exact location can be different. And I basically read everywhere that the result should be comparable.

Answer 1

I think you have hit the infamous scikit learn logistic regression defaults to regularisation of 1. https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

https://news.ycombinator.com/item?id=21416702

You can fix your simple problem by setting penalty='none' or C=100 in the constructor. C=100 would be necessary for older versions that defaulted to the liblinear algorithm.

# Logistic Regression
clf = LogisticRegression().fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 0)

clf = LogisticRegression().fit(X, y_or)
clf.predict(X) # --> array([1, 1, 1, 1)

#reduce regularisation by setting penalty='none' or inverse regularisation to 100
# Logistic Regression
clf = LogisticRegression(penalty='none').fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 1)
clf = LogisticRegression(C=100).fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 1)

clf = LogisticRegression(penalty='none').fit(X, y_or)
clf.predict(X) # --> array([0, 1, 1, 1)
```