0
$\begingroup$

To help me with some understanding, I'm trying to learn the Logical AND and Logical OR using Linear Regression trained over the following data:

import numpy as np
from sklearn.linear_model import LogisticRegression, Perceptron

X = np.array([
    [0, 0],
    [0, 1],
    [1, 0],
    [1, 1]
])

y_or = np.array([0, 1, 1, 1])
y_and = np.array([0, 0, 0, 1])

Intuitively, I don't see any problem as the data points are linearly separable w.r.t. to their class 0 or 1. For comparison, I also tried the Perceptron algorithm, which returns the correct predictions. However, the Linear Regression gets it wrong

# Logistic Regression
clf = LogisticRegression().fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 0)

clf = LogisticRegression().fit(X, y_or)
clf.predict(X) # --> array([1, 1, 1, 1)


# Perceptron
clf = Perceptron().fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 1])

clf = Perceptron().fit(X, y_or)
clf.predict(X) # --> array([0, 1, 1, 1])

I feel I'm missing something silly here. Or my understanding is simply way off. I would expect that Linear Regression and Perceptron have similar decision boundaries -- of course, the exact location can be different. And I basically read everywhere that the result should be comparable.

$\endgroup$
3
  • $\begingroup$ You may be fitting logistic regression without the needed interaction term for the two columns of predictor values. Once you add this interaction, logistic regression has to provide the right answer. $\endgroup$ Dec 22, 2021 at 14:05
  • 2
    $\begingroup$ I think you have hit the infamous scikit learn logistic regression defaults to regularisation of 1.ryxcommar.com/2019/08/30/scikit-learns-defaults-are-wrong $\endgroup$
    – seanv507
    Dec 22, 2021 at 17:00
  • $\begingroup$ so you have to set the C to a very large number and you should get unregularised logistic regression. I fixed your error with C=100. $\endgroup$
    – seanv507
    Dec 22, 2021 at 17:48

1 Answer 1

2
$\begingroup$

I think you have hit the infamous scikit learn logistic regression defaults to regularisation of 1. https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

https://news.ycombinator.com/item?id=21416702

You can fix your simple problem by setting penalty='none' or C=100 in the constructor. C=100 would be necessary for older versions that defaulted to the liblinear algorithm.

# Logistic Regression
clf = LogisticRegression().fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 0)

clf = LogisticRegression().fit(X, y_or)
clf.predict(X) # --> array([1, 1, 1, 1)

#reduce regularisation by setting penalty='none' or inverse regularisation to 100
# Logistic Regression
clf = LogisticRegression(penalty='none').fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 1)
clf = LogisticRegression(C=100).fit(X, y_and)
clf.predict(X) # --> array([0, 0, 0, 1)

clf = LogisticRegression(penalty='none').fit(X, y_or)
clf.predict(X) # --> array([0, 1, 1, 1)
```
$\endgroup$
4
  • 1
    $\begingroup$ I thought there was an option to turn off regularization. $\endgroup$
    – Dave
    Dec 22, 2021 at 17:54
  • 1
    $\begingroup$ @Dave thanks, updated $\endgroup$
    – seanv507
    Dec 22, 2021 at 18:07
  • 2
    $\begingroup$ See also stats.stackexchange.com/a/506332/232706 $\endgroup$ Dec 22, 2021 at 19:21
  • $\begingroup$ @seanv507 thanks! I tried a couple of different parameters (e.g., different solvers) but changing the strength of the regularization (or switching it completely off) didn't come to my mind. $\endgroup$
    – Christian
    Dec 23, 2021 at 0:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.