1
$\begingroup$

For linear regression, it is possible to have both continuous and dichotomous predictors. I assume this might have a similar rationale as the point-biseral correlation, where we have one dichotomous and one continuous variable.

I was wondering, however, why it doesn't work the other way round; why can't a linear regression be used to predict a dichotomous outcome from a continuous variable?

I am sure there is a logical explanation and I'd be curious to know more about this.

$\endgroup$
4
$\begingroup$

If I understand your question, then the answer is: "We can try to use a linear regression to predict a dichotomous variable...but should we?"

Your question basically gets to the heart of Generalized Linear Models. I won't cover that in detail here because it's discussed on the site extensively. I'll just add a brief, hopefully intuitive way to think about it.

A simple linear regression takes the form:

$$y_i = \beta x_i + \epsilon_i, \text{ with }\epsilon_i \sim N(0, \sigma^2)$$

Now imagine each $x_i$ is a positive, continuous number taking on large values (say square footage of a house), and each $y_i$ takes on values $1$ or $0$ (sold or not sold). You want to find the $\beta$ that fits this model the best using a linear regression.

The problem is that the quantity $\beta x_i$ is unbounded. So you might have estimates that are much greater than 1 or much smaller than 0. Say after performing maximum likelihood estimation, you get $\beta$ = 4.5. Say $x_1 = 1000$. What does it mean for $\hat{y_1} = 4500$? Do we assign that a value of $1$? Do we have more evidence for this house being sold than we do for a house with $\hat{y_2} = 2500$? How do we interpret $\beta$ in this case? You can certainly do this, and perhaps have a custom threshold such as:

$$ y_i = \begin{cases} 1 & \text{if } \beta x \geq 1000 \\ 0 & \text{if } \beta x < 1000 \end{cases} $$

But again, you lose something in the way of interpretability. As an aside, you get a function that isn't differentiable everywhere (because of the kinks at the extremes) which could make certain learning tasks tricky.

What we want instead is to encode this type of threshold apriori in the model, not after the fact like above. That is, we want a way to map values of $\beta x$ that are large to be close to 1 and values that are small to be close to 0. When I say large and small, I'm talking arbitrarily large or small.

To do this, you need to apply some transformation function (link) to our linear predictor, $\beta x$. This relates to Generalized Linear Models, of which there is a lot of information on this site.

Usually for dichotomous outcomes we might use a logit link, yielding the logistic regression. You've probably seen the sigmoid function before.

Instead of solving for $y_i$ (some potentially large or negative number) we solve for $p_i = P(y_i=1|x_i)$ - a probability that is between 0 and 1, reflecting what we want out of our dichotomous outcome.

The problem then becomes:

$$P(y_i=1|x_i) = p_i = \frac{1}{1+e^{-\beta x_i}}$$

Where we define:

$$\text{logit}(p_i) = \text{log}\big(\frac{p_i}{1-p_i}\big) = \beta x_i$$

So you see that we are reframing the question in terms of a probability instead of actual value. This is how we bound the output to be between 0 and 1, aligning with our intuition that a house can either be sold or not sold.

Note that you can reinterpret this to be a latent variable model, similar to the thresholding above:

$$ y_i = \begin{cases} 1 & \text{if } \beta x + \epsilon \geq 0 \\ 0 & \text{if } \beta x + \epsilon < 0 \end{cases} $$

Where $\epsilon$ follows a Logistic distribution. When $\epsilon$ follows a Normal distribution, we get a probit regression. To show this, you simply solve the formula for $\epsilon$ and use it's CDF to find the probability. The Logistic Distribution has a CDF that's effectively the same (given the right parameters) as our definition for $p_i$.

In general these transformations ensure that our outcomes will always be between 1 and 0, because of how they are defined. There are a number of beneficial properties that fall out of that, including the interpretability of the coefficients in this case, and statements such as "The probability of a house of size 1000 square feet getting sold is X%", something a linear regression can't give you. And now you can more easily compare houses against each other, with respect to their propensity to be sold.

Edit

I figured that a visual might help deliver the message. scikit-learn has an example that illustrates what we're talking about above. I present it below, in full, slightly modified to include this notion of a threshold:

import numpy as np
import matplotlib.pyplot as plt

from sklearn import linear_model

# this is our test set, it's just a straight line with some
# Gaussian noise
xmin, xmax = -5, 5
n_samples = 100
np.random.seed(0)
X = np.random.normal(size=n_samples)
y = (X > 0).astype(np.float)
X[X > 0] *= 4
X += .3 * np.random.normal(size=n_samples)

X = X[:, np.newaxis]
# run the classifier
clf = linear_model.LogisticRegression(C=1e5)
clf.fit(X, y)

# and plot the result
plt.figure(1, figsize=(10, 8))
plt.clf()
plt.scatter(X.ravel(), y, color='black', zorder=20, marker='x')
X_test = np.linspace(-5, 10, 300)


def sigmoid(x):
    return 1 / (1 + np.exp(-x))

loss = sigmoid(X_test * clf.coef_ + clf.intercept_).ravel()
plt.plot(X_test, loss, color='red', linewidth=2)

ols = linear_model.LinearRegression()
ols.fit(X, y)


def threshold(x):
    if ols.coef_ * x + ols.intercept_ >= 1:
        return 1
    elif ols.coef_ * x + ols.intercept_ <= 0:
        return 0
    return ols.coef_ * x + ols.intercept_

lin_pred = [threshold(i) for i in X_test]

plt.plot(X_test, lin_pred, linewidth=2)
plt.axhline(.5, color='.5')

plt.ylabel('y')
plt.xlabel('X')
plt.xticks(range(-5, 10))
plt.yticks([0, 0.5, 1])
plt.ylim(-.25, 1.25)
plt.xlim(-4, 10)
plt.legend(('Logistic Regression Model', 'Linear Regression with threshold'),
           loc="lower right", fontsize=14)
plt.title("Logistics Regression compared to Linear Regression\n for dichotomous outcome", size=16)
plt.show()

enter image description here

And finally, we can make predictions using both models - although only probabilistic statements with Logistic Regression. Hopefully it's readily obvious why Logistic Regression would be preferred to Linear Regression:

x = 3
print("When x=3...")
print("Linear Regression predicts:", threshold(x)[0])
print("Logistic Regression predicts:", int(clf.predict(x)[0]))
print()
x = 10
print("When x=10...")
print("Linear Regression predicts:", threshold(x))
print("Logistic Regression predicts:", int(clf.predict(x)[0]))

# When x=3...
# Linear Regression predicts: 0.757941284936912
# Logistic Regression predicts: 1

# When x=10...
# Linear Regression predicts: 1
# Logistic Regression predicts: 1

print("Probability that class = 1 when x = 0.2:", clf.predict_proba(0.2)[0][1])
print("Probability that class = 1 when x = 1.5:", clf.predict_proba(2.5)[0][1])

# Probability that class = 1 when x = 0.2: 0.4335699991588034
# Probability that class = 1 when x = 1.5: 0.9998355617824308
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.