Several short questions on non-linearity in multiple linear & logistic regression?

Question

I have a few related questions that have been bugging me for quite a while regarding non-linearity in linear & logistic regression with multiple predictors.

EDIT: I have since removed parts 3 and 4 of the question (will post separately).

1. Visualizing non-linearity in multiple linear/logistic regression

When building regression/classification predictive models with multiple predictors, one of the things I have never fully understood is if one can visually determine when a transformation is appropriate on the predictors.

It is clear when plotting $y \times x$ for simple linear regression where a relationship could be non-linear and a log/square-root/polynomial/spline transformation of $x$ can help model this non-linearity, but does this logic extend reliably to multiple regression? Could the observed non-linearity not be explained away by other predictors in the model?

Every text I read seems to only talk about non-linear transformations in the simple linear/logistic regression scenario, so it's not clear to me whether I can just extend this logic in the presence of other predictors and still expect model improvement. I guess an equivalent question but reversed would be "if a linear fit is best in the simple linear regression case, will it also be best in the presence of other predictors for multiple regression?"

For example, if I am building a multiple regression

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3$$

If I plot the relationship between $y$ and $x_3$ and think "hmm, this relationship is non-linear, perhaps I should add a second/third-order term for $x_3$ or use a spline basis with 4 knots", is it reasonable to assume this will also be a good transformation in multiple regression? Even if there are cases where this isn't the case, would you say it is still a reasonable strategy, or totally pointless?

2. Visualizing non-linearity (logistic, specifically)

Furthermore, if the above approach is reasonable, is there a similarly reliable way to visually determine non-linearity with the logit? I tried an approach for assessing linearity in logistic regression (could be misinformed) which involves binning numeric predictors before into equal-spaced bins, e.g. if we are fitting

$$ln \left(\frac{p}{1-p} \right) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3$$

I thought I could perhaps bin $x_1$ into equal-range bins, say $[0, 5), [5, 10), \dots, [25, 30)$, calculate $p$ as the proportion of each bin that is an 'event', then the log odds $ln \left(\frac{p}{1-p} \right)$. I would then plot the log odds across the bins to assess if linearity is reasonable.

The problem is I perceive this approach having the same issues as in the regression case in part 1. (if they are indeed issues), and the arbitrary selection of how wide the bins are changes how non-linear the relationship looks. Both of these together usually puts me off using this approach at all.

Answer 1

Good questions, but they do not have simple answers. When we have more than one predictor, things become much more complicated. Even more complicated when there is some correlation/relationship between the predictors.

Note that if $x_1$ and $x_2$ are moderately to strongly correlated, then there will a moderate to strong relationship between $x_1^2$ and $x_1 \times x_2$. Which means that your model may not be able to tell the difference between a quadratic relationship with $x_1$ and an interaction effect between $x_1$ and $x_2$.

Depending on the goal of your analysis, it may not matter which one you use (predictive modeling where you just want to predict a new case and your training data is representative of the population of interest). But other cases (causal inference, really understanding what leads to these relationships) will be very different between the models that the computer cannot distinguish between. In those cases you may need to depend on the science behind the data to decide what makes the most sense, or resort to more formal experimentation where you control the predictor values and remove the natural relationships.

As you have more predictors and more relationships between them it becomes likely that the simple 2 variable relationships will be different from the relationship when including multiple variables. You will need to be guided by the science behind the data and the goals of the analysis. There is a famous quote by Box: "All models are wrong, some models are useful". Whether you use polynomials, splines, etc. these are all approximations to some underlying truth. You need to use your knowledge and experience to determine what the models are telling you. Sometimes we fit things like splines, then look at the relationship and see that it looks like a particular transformation, then if that transformation makes sense with the science, refit the model using the transform.

For visualizing if the relationship is linear or not while correcting for other variables, do a search on the phrases "partial residual plot" and "added variable plot". These are primarily for linear regression, but with some practice (and enough data) they can also be suggestive for logistic regression models.

Your approach of binning a predictor is one approach to look for non-linearity, another approach is to fit a model that is linear in the predictor of interest, then refit with some type of curvature (splines are good, but not the only option) and compare the fits of the 2 models. You can use a formal full-reduced model test to compare them, but I prefer using tools like AIC, comparing the predictions, or other measures of goodness of fit to decide rather than the p-value from the full-reduced model test.

Your questions illustrate why statistics requires people with knowledge beyond a memorized formula who can do background research and reason out what makes sense.