13
$\begingroup$

I know that in linear regression the response variable must be continuous but why is this so? I cannot seem to find anything online that explains why I cannot use discrete data for the response variable.

$\endgroup$
25
$\begingroup$

There's nothing stopping you using linear regression on any two columns of numbers you like. There are times when it might even be a quite sensible choice.

However, the properties of what you get out won't necessarily be useful (e.g. won't necessarily be all you might want them to be).

Generally with regression you're trying to fit some relationship between the conditional mean of Y and the predictor -- i.e. fit relationships of some form $E(Y|x) = g(x)$; arguably modelling the behavior of the conditional expectation is what 'regression' is. [Linear regression is when you take one particular form for $g$]

For example, consider an extreme cases of discreteness, a response variable whose distribution is at either 0 or 1 and which takes the value 1 with probability that changes as some predictor ($x$) changes. That is $E(Y|x) = P(Y=1|X=x)$.

If you fit that sort of relationship with a linear regression model, then aside from a narrow interval, it will predict values for $E(Y)$ that are impossible -- either below $0$ or above $1$:

0-1 data and least squares fit

Indeed, it's also possible to see that as the expectation approaches the boundaries, the values must more and more frequently take the value at that boundary, so its variance gets smaller than if the expectation were near the middle -- the variance must decrease to 0. So an ordinary regression gets the weights wrong, underweighting the data in the region where the conditional expectation is near 0 or 1. SImilar effects occur if you have a variable bounded between a and b, say (such as each observation being a discrete count out of a known total possible count for that observation)

In addition, we normally expect the conditional mean to asymptote toward the upper and lower limits, which means the relationship would normally be curved, not straight, so our linear regression likely gets it wrong within the range of the data as well.

Similar issues occur with data that's only bounded on one side (e.g. counts that don't have an upper boundary) when you're near that one boundary.

It's possible (if rare) to have discrete data that's not bounded on either end; if the variable takes a lot of different values the discreteness may be of relatively little consequence as long as the model's description of the mean and the variance are reasonable.

Here's an example that it would be completely reasonable to use linear regression on:

plot showing discrete y as function of x where linear regression makes sense

Even though in any thin strip of x-values there's only a few different y-values that are likely to be observed (perhaps around 10 for intervals of width 1), the expectation can be well-estimated, and even standard errors and p-values and confidence intervals will all be more or less reasonable in this particular case. Prediction intervals will tend to work somewhat less well (because the non-normality will tend to have a more direct impact in that case)

--

If you want to perform hypothesis tests or calculate confidence or prediction intervals, the usual procedures make an assumption of normality. In some circumstances, that can matter. However, it's possible to inference without making that particular assumption.

$\endgroup$
  • $\begingroup$ Thank you, not sure I understood everything you have said but I will work on it. $\endgroup$ – ilovestats Mar 12 '17 at 8:11
  • 3
    $\begingroup$ If you have specific questions I can try to answer them $\endgroup$ – Glen_b -Reinstate Monica Mar 12 '17 at 9:12
  • $\begingroup$ @ilovestats I have an MA in Econometrics and I can assure you this answer is worth understanding every word of. Excellent answer, with an easy segue / good foundation to introduce logistic regression. $\endgroup$ – d8aninja Mar 12 '17 at 16:31
3
$\begingroup$

I can't comment, so I'll answer: in ordinary linear regression the response variable need not to be continuous, your assumption is not:

$$ y = β_0 + β_1x $$

but is:

$$ E[y] = β_0 + β_1x. $$

Ordinary linear regression derives from the minimization of the squared residuals, which is a method believed to be appropriate for continuous and discrete variables (see Gauss-Markof theorem). Of course generally used confidence or prediction intervals and hypothesis tests lay on normal distribution assumption, like Glen_b correctly pointed out, but OLS estimates of parameters does not.

$\endgroup$
2
$\begingroup$

In linear regression, the reason we need response to be continuous is combing from the assumptions we made. If the independent variable $x$ is continuous, then we assume the linear relationship between $x$ and $y$ is

$$y=\beta_0+\beta_1 x+\epsilon$$

where, the residual $\epsilon$ are normal. And form the formula we know $y$ is continuous.

On the other hand, in generalized linear model, the response variable can be discrete / categorical (logistic regression). Or count (Poisson regression).


Edit to address mark999 and remapt's comments.

Linear regression is a general term that may people use it differently. There is nothing to prevent us to use it on discrete variable OR the independent variable and dependent variable are not linear.

If we assume nothing and run linear regression, we still can get results. And if the results satisfy our needs, then the whole process is OK. However, As Glan_b said

If you want to perform hypothesis tests or calculate confidence or prediction intervals, the usual procedures make an assumption of normality.

I have this answer is because I assume OP is asking the linear regression from classical statistics book where we usually has this assumption when teach linear regression.

$\endgroup$
  • $\begingroup$ Thank you, I understood your explanation. Most appreciated. $\endgroup$ – ilovestats Mar 12 '17 at 8:13
  • 1
    $\begingroup$ Can you also explain why the explanatory variable can be either continuous or discrete (as many publications say)? In your explanation you say (and it makes sense) that the independent variable x is continuous. $\endgroup$ – ilovestats Mar 12 '17 at 8:34
  • 2
    $\begingroup$ I don't think this answer is correct. The response variable is not assumed to be a deterministic function of the explanatory variable(s), and there is no need to assume that the explanatory variable(s) is continuous. $\endgroup$ – mark999 Mar 12 '17 at 9:49
  • 2
    $\begingroup$ The outcome can be discrete or contionues, this answer is plain wrong $\endgroup$ – Repmat Mar 12 '17 at 12:36
  • $\begingroup$ @Repmat thanks for your comment, please check my edit. $\endgroup$ – Haitao Du Mar 12 '17 at 16:51
0
$\begingroup$

It doesn't. If the model works, who cares?

From a theoretic perspective the answers above are correct. However, in practical terms, it all depends on the domain of your data and the predictive power of your model.

One real-life example is the old MDS Bankruptcy Model. This was one of the early risk scores used by consumer credit lenders to predict the likelihood that a borrower would declare bankruptcy. This model used detailed data from the borrower's credit report and and a binary 0/1 flag to indicate bankruptcy over the prediction period. Then then fed that data into... yep.. you guessed it.

A Plain Old Linear Regression

I once got the opportunity to talk to one of the people who built this model. I asked him about the violation of assumptions. He explained that even though it completely violated the assumptions about residuals, etc. he didn't care.

Turns out...

This 0/1 linear regression model (when standardized/scaled to an easy-to-read score and paired with an appropriate cutoff) validated cleanly against holdout samples of data & performed very well as a Good/Bad discriminator for Bankruptcy.

The model was used for years as a 2nd credit score to guard against bankruptcy side by side with FICO's risk score (which was designed to predict 60+ days credit delinquency).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.