Choosing a better model based on the fit of the current model

Question

Suppose you are building a linear regression model. You have fit the model $Y = \beta_0 + \beta_1 X$. It doesn't fit well, so you want to consider a more complex model to improve the fit. Suppose again that you have two options for the more complex model; $Y = \beta_0 + \beta_1 f(X)$ or $Y = \beta_0 + \beta_1X + \beta_2Z$. In the first case $f(X)$ is a nonlinear function applied to X, and in the second case we just add a new predictor.

My question is, is there a heuristic for making this choice based on study of the residuals of the $Y = \beta_0 + \beta X$ fit? I know the answer might depend on the function f, but I'm more interested if someone can name a situation where there is evidence for which choice is better in the analysis of model fit.

Answer 1

Hard to say. What if $Z$ is strongly correlated with $f(X)$? As in, $X$ is length, $f(X)=X^3$ and $Z$ is weight. How could you tell? What would it mean?

However, if $Y$ is a non-linear function of $X$, but you fit $Y=a + bX$, then basically, all you are doing is tilting the function (and giving it a different origin). So if the true function is quadratic, the residuals are still going to look quadratic. The simple regression is not going to take any of the bumps or curves out of the original data. That should be a giveaway. This is especially true if the residuals appear to have the same variance (and my model is not improved by transforming $Y$ -- another interesting possibility).

But these are simply heuristics. Given a sequence $(X_i, Y_i)$ I can always find a function $f(X)$ to fit perfectly (an infinite number, actually), thus ruling out the need for $Z$.

Of course, if I had subject matter reasons for believing that $Z$ causes $Y$, I would include it in the model.