Does one really need to normalize the features of a regression model when doing R^2 explained variance analysis if regression is convex?

Question

TLDR; I want to know the percentage % of explained variance of the dependent variable given a list of D independent variables with crazy different scales -- but I believe that given convexity of regression theoretically we don't need to normalize features. Is this correct?

---

I have a dependent variable ($y$)and I want to understand how much each of my independent variables ($x_i$) explain the $y$'s variance (ideally as a percentage). This sounds to me as a standard R^2. However, I have some $x_i$'s are bounded between $[0,1]$ and others can have a value of 10 Billion (100B, 10_000_000_000). Due to convexity the solution should be easy to find (though I'm realizing that perhaps if we have more data points that features some subtleties of ill-posedness/none-uniqueness might come into place, but regardless, even the psuedo norm gives the min norm solution in all I cases I think, so I think it's still easy to find due to convexity). Ideally I'd try to collect as many number of points as possible & hopefully the hand crafted/chosen features I am choosing for the scientific study explain the variance.

So is it correct that due to convexity normalization is not important in terms of the statistical results? (I infer numerically it might matter but just statistically does it matter?)

---

Other thoughts

What about plotting y vs x?

Intuitively, what I am trying to see is how each $x_i$ explains $y$ -- or if it correlates with $x_i$. i.e. if $y$ is some function of $x_i$. One other way to do this is to plot the raw $y$'s vs $x_i$'s -- however, this doesn't scale beyond 2D point x's (perhaps 3D x's) -- but in my mind that would be nearly ideal way to plot the data because it wouldn't involve any numerical issues or statistical choices, we just see if $y$ really varies as $x_i$'s vary. It also, doesn't depend on the data because "nature" deals with the normalization implicitly/by itself when it computes $y = f(x)$ and we just try to plot the raw points. But of course this doesn't scale. Another way is to fix a subset of $x$'s and just plot 2 at a time. This idea come from trying to avoid the normalization of the features.

Synthetic data

One case I realized makes sense to normalize is with synthetic data. If I generate synthetic data where the $x_i$'s are of such difference -- then I must normalize the features of choose the coefficients very carefully such that the 10 billion scale features doesn't dominate the output just randomly. So normalization when creating synthetic data seems important -- at least as a thought experiment.

So when do I need to normalize my features? Does my use case need it? Theoretically is it actually needed when the problem is convex?

Answer 1

If you scale a feature in an OLS linear model, that is equivalent to multiplying the estimated coefficient by the reciprocal of that scaling factor. The following simulation is suggestive, and I think this makes sense when you consider the OLS estimator in its matrix form.

set.seed(2023)
N <- 1000
x1 <- runif(N, 0, 1)
x2 <- runif(N, 100000, 1000000)
x2_scaled <- x2/max(x2)
y <- x1 + x2 + rnorm(N)
L1 <- lm(y ~ x1 + x2)
L2 <- lm(y ~ x1 + x2_scaled)
summary(L1)$coef[3, 1] / (1/max(x2)) # 999251.6
summary(L2)$coef[3, 1]               # 999251.6

If you scale a variable, the inner workings of the OLS estimation will adjust the regression coefficient to make up for it. Consequently, while scaling variables might make their interpretation easier (e.g., a unit conversion or putting a variable in terms of standard deviations), this should not have any impact on the $R^2$.

If you're doing some other form of modeling, $R^2$ doesn't correspond to the proportion of variance explained by the regression. The links below get into why.

Interpreting nonlinear regression $R^2$

Why does regularization wreck orthogonality of predictions and residuals in linear regression?

Answer 2

While unstandardized regression coefficients are scale-dependent, $R^2$ is not dependent on the scale of the variables. This is part of why some people particularly favor $R^2$ (and $f^2$) for reporting effect sizes. It sounds like you are planning on computing $\Delta R^2$ for each variable, perhaps by running a regression with and without-each one? In that case, you don't need to scale your variables first. Note that the sum of the $\Delta R^2$s may not equal the $R^2$ of the full model (see here).