I've read a lot about overfitting here, and now I have a question about this subject.

Ok, if we put together too much independent variables, primary looking for better fits (for exemple, higher R², lower AIC, [...]), the model could be very unreal, or unable to predict something. The coefficients could have a difficult interpretation too (considering a ceteris paribus interpretation as a desirable thing).

That is, we don't really have a better model if it demands too many crucial information entered by the user (and much more objections).

Is it possible that data transformations could lead to some of that problems?

For exemple: I discover a better fit between $y$ and $x_i, i={1,2,3...}$, using $x_1$ and $x_6$ transformations like

$x_1._1 = x_1^{0.58}$ instead of tradicional $x_1._1 = x_1^{0.5} = sqrt(x_1)$

(after testing $0.57, 0.56, 0.55, [...];$ and $0.59, 0.6, 0.61, [...]$)


$x_6._1 = 2.6^{x_6}$ instead of $x_6._1 = e^{x_6} = 2.718^{x_6}$

(You can imagine other non-usual transformations)

I know that the interpretation will be like less "visual", but assuming that I have all the hypothesis verified and more precision, doing that should be considered as overfitting too?

The real variables could have a strange relation in the population, after all. Is that reasonable?


It depends on your goals. Almost always, the model with maximum "accuracy" is not the same as the model with the best/most convenient "interpretability". You ultimately have three options here:

  1. Build the most accurate model you can (hard to interpret)
  2. Build the most interpretable model you can (could be more accurate)
  3. Strike some balance, ie: "satisfice", between 1 and 2 (which is neither most interpretable nor the most accurate)

In applications where the quality of the prediction has large consequences - I'm thinking about my work in healthcare - I prefer to build the most accurate model possible, and test it rigorously, to deliver the best outcomes to our patients. For the users who want to understand how the model works, I often pull out representative samples (one tree in a random forest) or build a simpler model to approximate the more complex one (multivariate regression with simple interpretations).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.