6
$\begingroup$

I found one similar question on cross validated, but it was unanswered; my apologies if this has been answered.

I'm experimenting with feature interaction in a regression model I'm working on in R. My only concern right now is with building and scaling/centering the features.

Is there a "more statistically correct" means of scaling/centering numeric interaction features? The options I've thought through include:

  • option 1: Scale/center all numeric features, THEN calculate interaction values
    • Secondary question here... do I need to then re-scale/re-center those values?
  • option 2: Calculate interactions on all features, then scale/center it all at the same time

Well I had histograms to accompany the code below, but I don't have enough rep to post links to images.

Example of option 1:

# taking a small sample of "airquality" data
set.seed(2)
my_aq <- data.frame(airquality[sample(1:nrow(airquality), 100), ])

# create a scaled/centered version
my_aq_pp_scaler <- caret::preProcess(my_aq, method=c("center", "scale"))
my_aq_scaled <- predict(my_aq_pp_scaler, my_aq)


# computing interactions with pre-scaled data
denmat_prescaled <- as.data.frame(model.matrix(~ .^2 - 1, data=my_aq_scaled))
hist(denmat_prescaled$`Ozone:Solar.R`, col='light blue', main="Pre-interaction-scale: Not Rescaled") 

enter image description here

Then if I re-scale/re-center that, I'm left with this, which seems fine:

# 1) do I need to scale/center again?
    denmat_pp_scaler <- caret::preProcess(denmat_prescaled, method=c("center", "scale"))
    denmat_prescaled_scaled <- predict(denmat_pp_scaler, denmat_prescaled)
    hist(denmat_prescaled_scaled$`Ozone:Solar.R`, col='light pink', main="Pre-interaction-scale: Also Rescaled")

enter image description here

I think this looks like what I would want from a machine-learning/modeling perspective. So if I go with option 1, I'd likely rescale/recenter.

Example of option 2:

# postscaled - not scaling until AFTER interactions have been computed
denmat2 <- model.matrix(~ .^2 - 1, data=my_aq)
denmat2_pp_scaler <- caret::preProcess(denmat2, method=c("center", "scale"))
denmat_postscaled <- as.data.frame(predict(denmat2_pp_scaler, denmat2))
hist(denmat_postscaled$`Ozone:Solar.R`, col='light green', main="No Pre-scale: Just Post-interaction-scale")  

enter image description here

Question:

Is one of these methods more statistically sound than the other? Or is this one of those "it depends" type situations. I find it interesting to see how much of an impact these different methods have on the overall skew of the final values as well. That was not something I anticipated. If anyone could apply a more statistically rigorous explanation of which is better and why it does/doesn't matter, that would be awesome. Thank you!

edit:

The main model I'm using now is extreme gradient boosting (xgboost) with the objective set to "reg:linear" but I will likely also be trying lasso and ridge regression with glmnet.

edit2

Due to some upvotes, I now have enough rep to add my histogram images.

$\endgroup$
  • $\begingroup$ What type of regression are you doing with this data? Just glm, or other fancier models? $\endgroup$ – horaceT Oct 7 '17 at 17:43
  • $\begingroup$ Sorry, I should have added that. The main model I'm exploring right now is extreme gradient boosting (xgboost) with my objective set to "reg:linear" but I will likely try several other strategies as well such as ridge and lasso regression with glmnet. $\endgroup$ – TaylorV Oct 7 '17 at 18:05
  • $\begingroup$ GBM doesn't usually require input normalization. Your base/weak learner is a linear fit, it should be able to learn from raw. $\endgroup$ – horaceT Oct 7 '17 at 18:14
  • $\begingroup$ Ah I see, that makes sense. But then it will matter more for when I start working with glmnet on ridge/lasso, right? $\endgroup$ – TaylorV Oct 7 '17 at 18:20
  • $\begingroup$ Some algorithms require scaling inputs, such as neural net, but linear model doesn't care. I believe elastic net does. The only time you want to normalize inputs is when you want to compare coefficients. $\endgroup$ – horaceT Oct 7 '17 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.