In linear regression why does the response variable have to be continuous?

Question

I know that in linear regression the response variable must be continuous but why is this so? I cannot seem to find anything online that explains why I cannot use discrete data for the response variable.

Answer 1

There's nothing stopping you using linear regression on any two columns of numbers you like. There are times when it might even be a quite sensible choice.

However, the properties of what you get out won't necessarily be useful (e.g. won't necessarily be all you might want them to be).

Generally with regression you're trying to fit some relationship between the conditional mean of Y and the predictor -- i.e. fit relationships of some form $E(Y|x) = g(x)$; arguably modelling the behavior of the conditional expectation is what 'regression' is. [Linear regression is when you take one particular form for $g$, specifically one where $g$ is linear in the coefficients; it can include models where the relationship between the conditional mean of $Y$ and $x$ that is curved.]

For example, consider an extreme cases of discreteness, a response variable whose distribution is at either 0 or 1 and which takes the value 1 with probability that changes as some predictor ($x$) changes. That is $E(Y|x) = P(Y=1|X=x)$.

If you fit that sort of relationship with a linear regression model, then aside from a narrow interval, it will predict values for $E(Y)$ that are impossible -- either below $0$ or above $1$:

Indeed, it's also possible to see that as the expectation approaches the boundaries, the values must more and more frequently take the value at that boundary, so its variance gets smaller than if the expectation were near the middle -- the variance must decrease to 0. So an ordinary regression gets the weights wrong, underweighting the data in the region where the conditional expectation is near 0 or 1. Similar effects occur if you have a variable bounded between $a$ and $b$, say (such as each observation being a discrete count out of a known total possible count for that observation)

In addition, we normally expect the conditional mean to asymptote toward the upper and lower limits, which means the relationship would normally be curved, not straight, so our linear regression likely gets it wrong within the range of the data as well.

Similar issues occur with data that's only bounded on one side (e.g. counts bounded below by 0 but which don't have an upper boundary) when you're near that one boundary.

It's possible (if rare) to have discrete data that's not bounded on either end; if the variable takes a lot of different values the discreteness may be of relatively little consequence as long as the model's description of the mean and the variance are reasonable.

Here's an example that it would be completely reasonable to use linear regression on:

Even though in any thin strip of x-values there's only a few different y-values that are likely to be observed (perhaps around 10 for intervals of width 1), the expectation can be well-estimated, and the properties of standard errors and p-values and confidence intervals will all be more or less reasonable in this particular case. Prediction intervals will tend to work somewhat less well (because the non-normality will tend to have a more direct impact in that case)

--

If you want to perform hypothesis tests or calculate confidence or prediction intervals, the usual procedures make an assumption of normality. In some circumstances, that can matter. However, it's possible to perform inference and prediction without making that particular assumption if it might be an issue.

Answer 2

I can't comment, so I'll answer: in ordinary linear regression the response variable need not to be continuous, your assumption is not:

$$ y = β_0 + β_1x $$

but is:

$$ E[y] = β_0 + β_1x. $$

Ordinary linear regression derives from the minimization of the squared residuals, which is a method believed to be appropriate for continuous and discrete variables (see Gauss-Markof theorem). Of course generally used confidence or prediction intervals and hypothesis tests lay on normal distribution assumption, like Glen_b correctly pointed out, but OLS estimates of parameters does not.

Answer 3

In linear regression, the reason we need response to be continuous is combing from the assumptions we made. If the independent variable $x$ is continuous, then we assume the linear relationship between $x$ and $y$ is

$$y=\beta_0+\beta_1 x+\epsilon$$

where, the residual $\epsilon$ are normal. And form the formula we know $y$ is continuous.

On the other hand, in generalized linear model, the response variable can be discrete / categorical (logistic regression). Or count (Poisson regression).

---

Edit to address mark999 and remapt's comments.

Linear regression is a general term that may people use it differently. There is nothing to prevent us to use it on discrete variable OR the independent variable and dependent variable are not linear.

If we assume nothing and run linear regression, we still can get results. And if the results satisfy our needs, then the whole process is OK. However, As Glan_b said

If you want to perform hypothesis tests or calculate confidence or prediction intervals, the usual procedures make an assumption of normality.

I have this answer is because I assume OP is asking the linear regression from classical statistics book where we usually has this assumption when teach linear regression.

Answer 4

It doesn't. If the model works, who cares?

From a theoretic perspective the answers above are correct. However, in practical terms, it all depends on the domain of your data and the predictive power of your model.

One real-life example is the old MDS Bankruptcy Model. This was one of the early risk scores used by consumer credit lenders to predict the likelihood that a borrower would declare bankruptcy. This model used detailed data from the borrower's credit report and and a binary 0/1 flag to indicate bankruptcy over the prediction period. Then then fed that data into... yep.. you guessed it.

A Plain Old Linear Regression

I once got the opportunity to talk to one of the people who built this model. I asked him about the violation of assumptions. He explained that even though it completely violated the assumptions about residuals, etc. he didn't care.

Turns out...

This 0/1 linear regression model (when standardized/scaled to an easy-to-read score and paired with an appropriate cutoff) validated cleanly against holdout samples of data & performed very well as a Good/Bad discriminator for Bankruptcy.

The model was used for years as a 2nd credit score to guard against bankruptcy side by side with FICO's risk score (which was designed to predict 60+ days credit delinquency).

Answer 5

in linear regression in its most basic formulation, i.e. minimization of squared errors, the response variable can be discrete. for instance, the so called linear probability model is often fitted to binary outcomes such as predicting failures. Because linear regression output is on $(-\infty,+\infty)$ it is further converted into 0/1 by comparing the output to a threshold, such as 0.5. a lot of classification algorithms are actually similar, i.e. the produce continuous score which is then converted into discrete classes.

Answer 6

$$ \Large Y_i = \beta_0 + \beta_1 x_i + \epsilon_i \quad \text{ where } \quad \epsilon_i \sim \mathcal N\left(0, \sigma^2\right)$$

• Outcome $Y_i$ is a continuous numeric variable because $$\epsilon_i \sim \mathcal N\left(0, \sigma^2\right)$$ is a continous random variable.

Makes sense?