Clues that a problem is well suited for linear regression

Question

I am learning linear regression using Introduction to Linear Regression Analysis by Montgomery, Peck, and Vining. I'd like to choose a data analysis project.

I have the naive thought that linear regression is suitable only when one suspects that there are linear functional relationships between explanatory variables and the response variable. But not many real-world applications would seem to meet this criterion. Yet linear regression is so prevalent.

What facets of a project would an experienced statistician be thinking about if they were in my shoes, looking for a question+data that is well suited for linear regression.

Answer 1

I have the naive thought that linear regression is suitable only when one suspects that there are linear functional relationships between explanatory variables and the response variable. But not many real-world applications would seem to meet this criterion.

This is not a correct understanding of what is "linear" in "linear regression".

It is not the relationship between $y$ and the $x$'s that is assumed to be of linear form (though all elementary examples are likely to mislead you).

The "linear" refers to the model being linear in the parameters, and non-linear relationships between $y$ and some $x$ can certainly be modelled that way.

There's an example with a single predictor here, but curvilinear models are more often fitted as multiple regression, where several functions of a predictor (x variable, independent variable) may occur in the regression, and this allows a lot of flexibility. This includes polynomial regression, for example. See some discussion and examples here.

However, if we allow for the fact that predictors can be transformed in order to fit curved relationships, linearity in parameters does also correspond to linearity in those transformed predictors.

In addition, many problems are close to linear (at least over the range of values being considered), or are so noisy that any mild curvature is not discernible, and a variety of simple models for an increasing or a decreasing relationship might do -- and in that case a linear choice may be both adequate and the simplest to fit and understand.

What facets of a project would an experienced statistician be thinking about if they were in my shoes, looking for a question+data that is well suited for linear regression.

The only time I might look for a problem to apply regression to would be when I am trying to find a good example for teaching. When actually in the position of doing statistical work (rather than explaining or teaching it), I choose the methodology to suit the question of interest (and the characteristics of the data), rather than choosing the data to suit the method.

Imagine a carpenter, for example. The carpenter doesn't pick up a spokeshave and say "what can I use this on?". Rather, the carpenter has a problem to solve, and in considering the characteristics of the problem ("what am I trying to make?" and "what kind of wood am I using?" and so on...) particular tools may be more relevant than others. Sometimes the tools that are available may limit or guide the choices (if you don't have a spokeshave, you may have to make do with something else ... or you may just have to go buy a spokeshave).

However, let's assume that you have a pocket statistician helping you out and you're trying to find a problem suited to linear regression. Then they might suggest you consider various regression assumptions and when they matter. I'll mention a few things.

If you're simply interested in fitting a relationship between y and some univariate (possibly transformed) x most of the assumptions don't necessarily matter to you (the Gauss-Markov theorem may be of some relevance). You'd be looking for a case where you think $E(y|g(x))$ is approximately linear in $g(x)$ for some - known - $g$ (that is, it assumes we know the functional form of relationship we want). Writing $x^*=x$, we need that $E(y|x^*)=a+bx*$ is at least approximately true.

If you're able to use multiple regression even that's not especially a major issue, since one can use (for example) cubic regression splines to fit fairly general relationships.

I'd suggest you steer clear of data over time unless you understand the issues with spurious regression; stick with cross-sectional problems.

If you're dealing with only a single $x$ I expect you want a continuous rather than categorical $x$.

You'd want not to have measurement error in the $x$ unless you're interested in the expectation conditioning on the measured value.

If you're interested in hypothesis testing, confidence intervals or prediction intervals, then more of the usual regression assumptions may matter (but there are alternatives that don't make those assumptions, and in some cases, at least some of the assumptions may not be particularly important anyway).

So one thing to at least try to be aware of is what those assumptions are that are made in deriving the inferential procedures you're using and how important they may be in your particular problem (as an example, when performing the usual hypothesis tests, normality is an assumption, but in large samples that assumption may not be important; on the other hand, the assumption of constant variance may be more of an issue).

There are a number of posts that discuss assumptions of regression, and some posts that discuss when they need to be made at all, and how much they might matter, and even what order to consider them in.

Answer 2

In addition to the excellent answers above, there are general requirements for the linear model to work reasonably well, mainly related to $Y$. $Y$ needs to be well behaved in the sense of not having extreme values that will overly influence the model fit. Secondly, $Y$ needs to luckily be transformed so that the model has a hope of being additive and so that residuals are Gaussian (if doing inference). Analysts frequently make the mistake of trying more than 2 transformations of $Y$ to satisfy model assumptions, which distorts the final inference. A simpler way to say this is we need to already understand the $Y$ distribution (conditional on $X$) well. Over many years of experience you'll see that certain variables such as blood pressure tend to behave well in a linear model and others (e.g., blood chemistry measurements) don't.

All this is in contrast to semiparametric models that only assume $Y$ is ordinal, are completely robust to strange values, and don't care about how $Y$ is transformed. Proportional odds and proportional hazards models are two example classes of models.

Answer 3

@Glen_b gave a very good answer but, as noted, didn't get to finish.

So, as to your last question:

An experienced statistician, I think, would not ask this question. As Glen notes, the problem dictates the tools to use, not the other way around.

If I were trying to learn a technique like linear regression I would use already worked examples - but ones that had real data, not made up data designed to make things easy. A book such as Regression Modelling by Example may provide guidance.

However, one of the first steps in looking at a regression problem is deciding whether linear regression is, in fact, suitable.

Answer 4

Many responses have touched on the assumptions that need to be met: linearity in the residuals, homogeneity of variance across the range of the predictor, no extreme values that could influence the regression line, and independent observations. Residual plots are fairly easy to produce with most regression programs and some packages provide some automatically (SAS).

One person talked about transforming y. This is common practice in some areas, but it is a practice that leads to biased and possibly un-interpretable results. The bias shows up when you try to back transform the results into the original metric. Better to shift to another type of regression that has a residual pattern that matches the distributional assumptions of the residual. See chapter 3 in Agresti's Introduction to Categorical Data Analysis where he introduces the concept of links. A number of regression textbooks also introduce the generalized linear model.