Can data transformations lead to overfitting?

Ok, if we put together too much independent variables, primary looking for better fits (for example, higher R², lower AIC, [...]), the model could be very unreal, or unable to predict something. The coefficients could have a difficult interpretation too (considering a ceteris paribus interpretation as a desirable thing).

That is, we don't really have a better model if it demands too many crucial information entered by the user (and much more objections).

Is it possible that data transformations could lead to some of that problems?

For example: I discover a better fit between $$y$$ and $$x_i, i={1,2,3...}$$, using $$x_1$$ and $$x_6$$ transformations like

$$x_1._1 = x_1^{0.58}$$ instead of traditional $$x_1._1 = x_1^{0.5} = sqrt(x_1)$$

(after testing $$0.57, 0.56, 0.55, [...];$$ and $$0.59, 0.6, 0.61, [...]$$)

or

$$x_6._1 = 2.6^{x_6}$$ instead of $$x_6._1 = e^{x_6} = 2.718^{x_6}$$

(You can imagine other non-usual transformations)

I know that the interpretation will be like less "visual", but assuming that I have all the hypothesis verified and more precision, doing that should be considered as overfitting too?

The real variables could have a strange relation in the population, after all. Is that reasonable?

• Sounds like a multiple testing problem (en.wikipedia.org/wiki/Multiple_comparisons_problem).
– Alex
Commented May 21, 2018 at 20:33
• In addition to the answer of Tim: there are some fields of study (mostly physics related) where you KNOW which transform applies. E.g. Heat transfer rate scales with 1/thickness, the position of the fluid front in capillary flow travels proportional to sqrt(time), and for reaction rates we have Arrhenius which implies logarithmic transform applies. In those cases (rare in some fields but common in others), you degrade your model by adjusting the transform as if it is a free parameter. For the rest, I agree with Tim.
– W_vH
Commented Dec 16, 2022 at 8:47

You are doing it wrong. If you need to find out an appropriate weights like $$x^\alpha$$ or $$\gamma^x$$, than those should be parameters of the model, rather than something you tune during hyperparameter tuning.

First of all, if you just tune them using a fixed grid (0.50, 0.51, 0.52, ...) you are missing the intermediate values, so you won't be able to find the values that might work better than the ones you checked.

Second, how would those values differ from parameters of a model? It shouldn't really matter if you transform the data with $$x^{0.51}$$ vs $$x^{0.52}$$, it would matter if you ask about square root vs squared values, but the model has parameters that would enable it to accommodate to the data and you should not need to pick the transformations by hand.

If you are seeing performance difference, than it is likely that it's either numerical issues (so you should rather do something like normalization of the data) or the differences are by chance (random initialization). In both cases, using something like $$k$$-fold cross-validation would be strongly encouraged to verify if the results are stable across different subsamples and runs.

It depends on your goals. Almost always, the model with maximum "accuracy" is not the same as the model with the best/most convenient "interpretability". You ultimately have three options here:

1. Build the most accurate model you can (hard to interpret)
2. Build the most interpretable model you can (could be more accurate)
3. Strike some balance, ie: "satisfice", between 1 and 2 (which is neither most interpretable nor the most accurate)

In applications where the quality of the prediction has large consequences - I'm thinking about my work in healthcare - I prefer to build the most accurate model possible, and test it rigorously, to deliver the best outcomes to our patients. For the users who want to understand how the model works, I often pull out representative samples (one tree in a random forest) or build a simpler model to approximate the more complex one (multivariate regression with simple interpretations).