Suppose you are building a linear regression model. You have fit the model $Y = \beta_0 + \beta_1 X$. It doesn't fit well, so you want to consider a more complex model to improve the fit. Suppose again that you have two options for the more complex model; $Y = \beta_0 + \beta_1 f(X)$ or $Y = \beta_0 + \beta_1X + \beta_2Z$. In the first case $f(X)$ is a nonlinear function applied to X, and in the second case we just add a new predictor.
My question is, is there a heuristic for making this choice based on study of the residuals of the $Y = \beta_0 + \beta X$ fit? I know the answer might depend on the function f, but I'm more interested if someone can name a situation where there is evidence for which choice is better in the analysis of model fit.