I found one similar question on cross validated, but it was unanswered; my apologies if this has been answered.
I'm experimenting with feature interaction in a regression model I'm working on in R. My only concern right now is with building and scaling/centering the features.
Is there a "more statistically correct" means of scaling/centering numeric interaction features? The options I've thought through include:
- option 1: Scale/center all numeric features, THEN calculate interaction values
- Secondary question here... do I need to then re-scale/re-center those values?
- option 2: Calculate interactions on all features, then scale/center it all at the same time
Well I had histograms to accompany the code below, but I don't have enough rep to post links to images.
Example of option 1:
# taking a small sample of "airquality" data
set.seed(2)
my_aq <- data.frame(airquality[sample(1:nrow(airquality), 100), ])
# create a scaled/centered version
my_aq_pp_scaler <- caret::preProcess(my_aq, method=c("center", "scale"))
my_aq_scaled <- predict(my_aq_pp_scaler, my_aq)
# computing interactions with pre-scaled data
denmat_prescaled <- as.data.frame(model.matrix(~ .^2 - 1, data=my_aq_scaled))
hist(denmat_prescaled$`Ozone:Solar.R`, col='light blue', main="Pre-interaction-scale: Not Rescaled")
Then if I re-scale/re-center that, I'm left with this, which seems fine:
# 1) do I need to scale/center again?
denmat_pp_scaler <- caret::preProcess(denmat_prescaled, method=c("center", "scale"))
denmat_prescaled_scaled <- predict(denmat_pp_scaler, denmat_prescaled)
hist(denmat_prescaled_scaled$`Ozone:Solar.R`, col='light pink', main="Pre-interaction-scale: Also Rescaled")
I think this looks like what I would want from a machine-learning/modeling perspective. So if I go with option 1, I'd likely rescale/recenter.
Example of option 2:
# postscaled - not scaling until AFTER interactions have been computed
denmat2 <- model.matrix(~ .^2 - 1, data=my_aq)
denmat2_pp_scaler <- caret::preProcess(denmat2, method=c("center", "scale"))
denmat_postscaled <- as.data.frame(predict(denmat2_pp_scaler, denmat2))
hist(denmat_postscaled$`Ozone:Solar.R`, col='light green', main="No Pre-scale: Just Post-interaction-scale")
Question:
Is one of these methods more statistically sound than the other? Or is this one of those "it depends" type situations. I find it interesting to see how much of an impact these different methods have on the overall skew of the final values as well. That was not something I anticipated. If anyone could apply a more statistically rigorous explanation of which is better and why it does/doesn't matter, that would be awesome. Thank you!
edit:
The main model I'm using now is extreme gradient boosting (xgboost) with the objective set to "reg:linear" but I will likely also be trying lasso and ridge regression with glmnet.
edit2
Due to some upvotes, I now have enough rep to add my histogram images.