A fundamental question about multivariate regression

Question

This is slightly embarrassing, as I've done a fair amount of statistical work, but for years I've heard this niggling voice at the back of my head, and I need to ask someone.

I remember when I first heard about multivariate regression and an example being used was a cancer trial of patients receiving surgery vs. chemotherapy (if I recall). Initially it appeared there was an advantage of one arm over another in terms of survival (days until death), but it turned out that there was a confounding covariate (e.g. age was much higher in one arm), and when this was factored in, there was no survival advantage for each surgery or chemotherapy.

When I've done covariate analysis, you obviously get a regression estimate for each variable, with the others taken into account.

I have a few questions based on the above examples

What would happen if you supplied two co-variates to the example above, the first being age in years, and the second being number of candles on your birthday cake.

Some people wont have any candles on their birthday cake, but lets say that generally people had more when they got older, and it was highly (but not quite perfectly) correlated.

Now candles aren't causative here, but would they falsely come out as having an adverse effect on mortality here? I know without age in the model, they would, but with age included, wouldn't they still? If not, that implies the model is able to ascertain that 'age' is to blame and 'candles' are not, which is obviously not possible for a statistical test. If so, do they 'share' some of the predictive power, and make 'age' look less important? Isn't this wrong? Is there some particular practice we should stick to to ensure we aren't including stuff like candles into models?

And a related example to the above, what would happen if we supplied not only the age in years, but a second parameter which was the age in days? If we supply more and more correlated columns, what happens?

Answer 1

First, a matter of terminology. According to present usage, "multivariate" should mean having several outcome variables rather than having multiple predictors. What you are describing is an example of Cox multiple (not "multivariate") regression. (I have erred in this usage myself.)

Second, your scenario is at the heart of the issue of feature selection, a topic with 1200 tagged qeustions on this site as I write. In real-world applications some predictors are typically correlated with each other. (See the 510 questions with the multicollinearity tag on this site.) The problem of how to attribute predictive power to individual variables necessarily arises in such analyses.

Third, your question also gets to the difference between explanation and prediction in models. Your asking about what is "causative" shows an interest in the former, but as you recognize this is difficult with correlated predictors. Nevertheless there are ways to try to approach causality with careful approaches involving manipulations and modeling; this page is an introduction to the combinations of issues in feature selection and causal inference.

If your interest is in prediction, however, correlated variables can provide an advantage instead of a problem--maybe not in this extreme example of perfect correlations with age, but with other predictors in clinical applications that are somewhat correlated. If each of your predictors may provide some information based on your understanding of the subject matter, there is no reason arbitrarily just to throw some predictors away. Typically, keeping more predictors will improve performance on new data sets. If there are large numbers of predictors, ridge regression will tend to treat correlated variables together while penalizing their regression coefficients to minimize risks of over-fitting.

Finally, I caution you against using unpenalized automated variable selection as with R's step functions. These might fit a particular data sample well but will often fail on new samples from the same population, as you can check by repeating modeling on multiple bootstrap samples of the same data set. Even principled penalized methods for feature selection like LASSO, which can provide good predictive performance, will provide inconsistent sets of selected predictors from data sample to data sample. So one must avoid the temptation to invoke a successful set of selected predictor variables as those "responsible for" the outcomes; that is, don't confuse prediction with explanation.

Answer 2

A similar question has been discussed extensively in this website, and elsewhere on the internet. Just look for "multicollinearity in regression" or "correlated predictors". Couple examples:

What is the effect of having correlated predictors in a multiple regression model?
Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?

In short, correlation between predictors does not bias the estimates, but the precision of the estimates goes down. Given perfect collinearity, the model would have no way of distinguishing the effects. Including 'candles' could potentially make the 'age' predictor appear insignificant (because of higher SE associated with the effect estimate).

Ideally, covariates should be selected based on prior knowledge about the causal connections. Of course, that is not always possible. In epidemiological settings I'd prefer to err on the side of caution by including more covariates - this way, even though you might lose some nice p-values, at least you will not get false associations because of omitted confounders. And with subset selection you might end up suggesting that birthday candles lead to death...