12
$\begingroup$

I am learning linear regression using Introduction to Linear Regression Analysis by Montgomery, Peck, and Vining. I'd like to choose a data analysis project.

I have the naive thought that linear regression is suitable only when one suspects that there are linear functional relationships between explanatory variables and the response variable. But not many real-world applications would seem to meet this criterion. Yet linear regression is so prevalent.

What facets of a project would an experienced statistician be thinking about if they were in my shoes, looking for a question+data that is well suited for linear regression.

$\endgroup$
  • 3
    $\begingroup$ I appreciate that you are learning a technique and want to know where it will work. But for experienced statisticians (and statistically minded scientists) the situation is entirely the other way round: there is a problem and data and then the question is what kind of models and methods are the best choice. You will get to see that linear regression as first encountered is just one flavour; with experience people are happy to jump to Poisson regression, logit regression, etc. and even linearity in parameters can easily be combined with more general structures. $\endgroup$ – Nick Cox Oct 15 '15 at 8:14
  • $\begingroup$ and of course time series mdels when the observations are potentially autocorrelated $\endgroup$ – IrishStat Oct 15 '15 at 12:03
  • 3
    $\begingroup$ In reality the linear models are used left and right even when we know that the relationship is nonlinear. Think of a linear model as a first order approximation, sort of a multivariate Taylor expansion. $\endgroup$ – Aksakal Oct 15 '15 at 13:14
12
$\begingroup$

I have the naive thought that linear regression is suitable only when one suspects that there are linear functional relationships between explanatory variables and the response variable. But not many real-world applications would seem to meet this criterion.

This is not a correct understanding of what is "linear" in "linear regression".

It is not the relationship between $y$ and the $x$'s that is assumed to be of linear form (though all elementary examples are likely to mislead you).

The "linear" refers to the model being linear in the parameters, and non-linear relationships between $y$ and some $x$ can certainly be modelled that way.

There's an example with a single predictor here, but curvilinear models are more often fitted as multiple regression, where several functions of a predictor (x variable, independent variable) may occur in the regression, and this allows a lot of flexibility. This includes polynomial regression, for example. See some discussion and examples here.

However, if we allow for the fact that predictors can be transformed in order to fit curved relationships, linearity in parameters does also correspond to linearity in those transformed predictors.

In addition, many problems are close to linear (at least over the range of values being considered), or are so noisy that any mild curvature is not discernible, and a variety of simple models for an increasing or a decreasing relationship might do -- and in that case a linear choice may be both adequate and the simplest to fit and understand.

What facets of a project would an experienced statistician be thinking about if they were in my shoes, looking for a question+data that is well suited for linear regression.

The only time I might look for a problem to apply regression to would be when I am trying to find a good example for teaching. When actually in the position of doing statistical work (rather than explaining or teaching it), I choose the methodology to suit the question of interest (and the characteristics of the data), rather than choosing the data to suit the method.

Imagine a carpenter, for example. The carpenter doesn't pick up a spokeshave and say "what can I use this on?". Rather, the carpenter has a problem to solve, and in considering the characteristics of the problem ("what am I trying to make?" and "what kind of wood am I using?" and so on...) particular tools may be more relevant than others. Sometimes the tools that are available may limit or guide the choices (if you don't have a spokeshave, you may have to make do with something else ... or you may just have to go buy a spokeshave).

However, let's assume that you have a pocket statistician helping you out and you're trying to find a problem suited to linear regression. Then they might suggest you consider various regression assumptions and when they matter. I'll mention a few things.

If you're simply interested in fitting a relationship between y and some univariate (possibly transformed) x most of the assumptions don't necessarily matter to you (the Gauss-Markov theorem may be of some relevance). You'd be looking for a case where you think $E(y|g(x))$ is approximately linear in $g(x)$ for some - known - $g$ (that is, it assumes we know the functional form of relationship we want). Writing $x^*=x$, we need that $E(y|x^*)=a+bx*$ is at least approximately true.

If you're able to use multiple regression even that's not especially a major issue, since one can use (for example) cubic regression splines to fit fairly general relationships.

I'd suggest you steer clear of data over time unless you understand the issues with spurious regression; stick with cross-sectional problems.

If you're dealing with only a single $x$ I expect you want a continuous rather than categorical $x$.

You'd want not to have measurement error in the $x$ unless you're interested in the expectation conditioning on the measured value.

If you're interested in hypothesis testing, confidence intervals or prediction intervals, then more of the usual regression assumptions may matter (but there are alternatives that don't make those assumptions, and in some cases, at least some of the assumptions may not be particularly important anyway).

So one thing to at least try to be aware of is what those assumptions are that are made in deriving the inferential procedures you're using and how important they may be in your particular problem (as an example, when performing the usual hypothesis tests, normality is an assumption, but in large samples that assumption may not be important; on the other hand, the assumption of constant variance may be more of an issue).

There are a number of posts that discuss assumptions of regression, and some posts that discuss when they need to be made at all, and how much they might matter, and even what order to consider them in.

$\endgroup$
  • $\begingroup$ Nice answer, but I think it does not answer the question completely. What facets of a project would an experienced statistician be thinking about if they were in my shoes, looking for a question+data that is well suited for linear regression. remains unanswered. $\endgroup$ – Dawny33 Oct 15 '15 at 7:34
  • $\begingroup$ @Dawny33 I definitely intend to add more to this later - some stuff came up while I was typing that which prevented me writing the full answer I originally intended; I had time only to finish the sentence I was on, and now may not get back to it for a day or two. Indeed I didn't even have time to correct all the typos in it. (In the meantime you should not hesitate to post an answer.) On the other hand, pointing out that the premise of the question is flawed may lead to the OP wanting to ask different things than they originally intended (it's often the case when the central premise fails) $\endgroup$ – Glen_b Oct 15 '15 at 9:37
  • $\begingroup$ For example, I anticipate one new question that could arise would be "do you have an example?". $\endgroup$ – Glen_b Oct 15 '15 at 9:57
  • $\begingroup$ @Glen_b thanks. The "linear" refers to the model being linear in the parameters. Sorry if I mis-wrote, I didn't mean to imply otherwise. The keyword was functional. $\endgroup$ – cwackers Oct 16 '15 at 5:01
  • $\begingroup$ @Glen_b they might suggest you consider various regression assumptions. Agreed again. I was not explicit about this, but my Q is more about domain knowledge. I'm wondering what an experienced statistician would be looking for in the system being being considered for LR analysis, hence my naive proposal of there being regressors that are linearly and functionally related to the response, and whose simultaneous relationship to the response is additive. $\endgroup$ – cwackers Oct 16 '15 at 5:16
4
$\begingroup$

In addition to the excellent answers above, there are general requirements for the linear model to work reasonably well, mainly related to $Y$. $Y$ needs to be well behaved in the sense of not having extreme values that will overly influence the model fit. Secondly, $Y$ needs to luckily be transformed so that the model has a hope of being additive and so that residuals are Gaussian (if doing inference). Analysts frequently make the mistake of trying more than 2 transformations of $Y$ to satisfy model assumptions, which distorts the final inference. A simpler way to say this is we need to already understand the $Y$ distribution (conditional on $X$) well. Over many years of experience you'll see that certain variables such as blood pressure tend to behave well in a linear model and others (e.g., blood chemistry measurements) don't.

All this is in contrast to semiparametric models that only assume $Y$ is ordinal, are completely robust to strange values, and don't care about how $Y$ is transformed. Proportional odds and proportional hazards models are two example classes of models.

$\endgroup$
  • $\begingroup$ Thanks for pointing out the aspect of being well-behaved. I've thought of transformations of the regressors, but not of the response variable. However, I see now how the later could be used to reshape the distribution of residuals. Thanks for filling in some of the picture. A very helpful post. $\endgroup$ – cwackers Oct 16 '15 at 5:28
3
$\begingroup$

@Glen_b gave a very good answer but, as noted, didn't get to finish.

So, as to your last question:

An experienced statistician, I think, would not ask this question. As Glen notes, the problem dictates the tools to use, not the other way around.

If I were trying to learn a technique like linear regression I would use already worked examples - but ones that had real data, not made up data designed to make things easy. A book such as Regression Modelling by Example may provide guidance.

However, one of the first steps in looking at a regression problem is deciding whether linear regression is, in fact, suitable.

$\endgroup$
  • $\begingroup$ An experienced statistician, I think, would not ask this question. yes, that why I qualified my Q with "in my shoes". Thanks so much for the book recommendation. I'll track down a copy. A slew of examples would help at least as half of the story, with counter examples being the other half. $\endgroup$ – cwackers Oct 16 '15 at 5:33
  • $\begingroup$ Ahhh, a quote! from page 2 of the 4th edition: We invite the readers to think about questions (in their own areas of work, research or interest) that can be addressed using regression analysis. $\endgroup$ – cwackers Oct 16 '15 at 5:54
0
$\begingroup$

Many responses have touched on the assumptions that need to be met: linearity in the residuals, homogeneity of variance across the range of the predictor, no extreme values that could influence the regression line, and independent observations. Residual plots are fairly easy to produce with most regression programs and some packages provide some automatically (SAS).

One person talked about transforming y. This is common practice in some areas, but it is a practice that leads to biased and possibly un-interpretable results. The bias shows up when you try to back transform the results into the original metric. Better to shift to another type of regression that has a residual pattern that matches the distributional assumptions of the residual. See chapter 3 in Agresti's Introduction to Categorical Data Analysis where he introduces the concept of links. A number of regression textbooks also introduce the generalized linear model.

$\endgroup$
  • $\begingroup$ I don't share the pessimism about transformation. After all the original transformation is quite arbitrary. If you transform and get residuals with a symmetric distribution, the back-transform of the predicted values is the predicted median on the original scale. Predicted medians are quite useful. If you want to get predicted means on the original scale you can use the smearing estimator. $\endgroup$ – Frank Harrell Oct 22 '15 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.