# Different results for Logistic Regression (wrong) and Perceptron (correct)

To help me with some understanding, I'm trying to learn the Logical AND and Logical OR using Linear Regression trained over the following data:

import numpy as np
from sklearn.linear_model import LogisticRegression, Perceptron

X = np.array([
[0, 0],
[0, 1],
[1, 0],
[1, 1]
])

y_or = np.array([0, 1, 1, 1])
y_and = np.array([0, 0, 0, 1])


Intuitively, I don't see any problem as the data points are linearly separable w.r.t. to their class 0 or 1. For comparison, I also tried the Perceptron algorithm, which returns the correct predictions. However, the Linear Regression gets it wrong

# Logistic Regression
clf = LogisticRegression().fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 0)

clf = LogisticRegression().fit(X, y_or)
clf.predict(X) # --> array([1, 1, 1, 1)

# Perceptron
clf = Perceptron().fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 1])

clf = Perceptron().fit(X, y_or)
clf.predict(X) # --> array([0, 1, 1, 1])


I feel I'm missing something silly here. Or my understanding is simply way off. I would expect that Linear Regression and Perceptron have similar decision boundaries -- of course, the exact location can be different. And I basically read everywhere that the result should be comparable.

• You may be fitting logistic regression without the needed interaction term for the two columns of predictor values. Once you add this interaction, logistic regression has to provide the right answer. Commented Dec 22, 2021 at 14:05
• I think you have hit the infamous scikit learn logistic regression defaults to regularisation of 1.ryxcommar.com/2019/08/30/scikit-learns-defaults-are-wrong Commented Dec 22, 2021 at 17:00
• so you have to set the C to a very large number and you should get unregularised logistic regression. I fixed your error with C=100. Commented Dec 22, 2021 at 17:48

I think you have hit the infamous scikit learn logistic regression defaults to regularisation of 1. https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

https://news.ycombinator.com/item?id=21416702

You can fix your simple problem by setting penalty='none' or C=100 in the constructor. C=100 would be necessary for older versions that defaulted to the liblinear algorithm.

# Logistic Regression
clf = LogisticRegression().fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 0)

clf = LogisticRegression().fit(X, y_or)
clf.predict(X) # --> array([1, 1, 1, 1)

#reduce regularisation by setting penalty='none' or inverse regularisation to 100
# Logistic Regression
clf = LogisticRegression(penalty='none').fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 1)
clf = LogisticRegression(C=100).fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 1)

clf = LogisticRegression(penalty='none').fit(X, y_or)
clf.predict(X) # --> array([0, 1, 1, 1)
$$$$
`
• I thought there was an option to turn off regularization.
– Dave
Commented Dec 22, 2021 at 17:54
• @Dave thanks, updated Commented Dec 22, 2021 at 18:07
• Commented Dec 22, 2021 at 19:21
• @seanv507 thanks! I tried a couple of different parameters (e.g., different solvers) but changing the strength of the regularization (or switching it completely off) didn't come to my mind. Commented Dec 23, 2021 at 0:41