3
$\begingroup$

Harrell's course notes for Regression Modeling Strategies state

One problem prevents most of these methods [mostly referring to lasso and relatives and variations] from being ready for everyday use: they require scaling predictors before fitting the model. When a predictor is represented by nonlinear basis functions, the scaling recommendations in the literature are not sensible. [section 2.5.1, at least in the 2016 version]

Is there a general problem with regularization and using multiple transformations of the same variable? Is there a full or partial solution (i.e., is there a "sensible" scaling recommendation)? And is there a similar scaling issue with interactions?

$\endgroup$

2 Answers 2

3
$\begingroup$

I can't get into @FrankHarrel 's mind, and do hope he offers an answer himself, but I'll add my own thoughts on your questions.

Is there a general problem with regularization and using multiple transformations of the same variable?

No, definitely not if your goal is prediction. In general, including transformations of a single variable makes inferential claims based on a model more difficult to make and support, but so does using a regularized estimator.

But there is some conceptual difficulties to be aware of when using standard regularization techniques along with basis expansions. When we use basis expansions in our regression our goal is to choose one functional form from an entire vector space of possibilities. For example:

  • Polynomial regression chooses the best fit from the vector space of polynomials with fixed degree.
  • Cubic spline regression chooses the best fit from the vector space of cubic splines with fixed knots.

For these methods to be conceptually grounded, it seems sensible to seek methods which are basis independent. This means that, while the implementation of our methodology may depend on picking a fixed basis of this vector space, the final fit model should not depend on our choice of basis.

As as example, consider the method of smoothing splines. In this method we take our vector space of functional forms as the space of all continuous, second differentiable functions. We fit the curve by minimizing the following loss function:

$$L(y, f) = \sum_i (y_i - f(x_i))^2 + \lambda \int (f''(t))^2 dt$$

This loss function is clearly basis independent by inspection as it only references the functions $f$ themselves, not their expression in a basis.

The regularization terms is conceptually chosen, we penalize the fit to the data according to how "wiggly" the fit function is. I.e., we prefer fits that wiggle less.

To fit the model, we do need to choose a basis $f_1, f_2, f_3, \ldots$. If we then expand the expression of the loss function in this basis by writing a general $f$ in terms of our chosen basis,

$$ f = \sum_j \alpha_j f_j$$

the squared error term is the usual business, and the regularization term becomes:

$$ \lambda \alpha^t \Omega \alpha $$

where $\Omega$ is a matrix of inner products of second derivatives

$$ \Omega_{pq} = \int f''_p(t) f''_q(t) dt $$

Now, if we use standard regularized regression using this basis, we would instead get a regularization term like

$$ \lambda \alpha^t \alpha $$

This is not basis independent. We penalize wiggly and non-wiggly basis elements the same amount, so we have lost some conceptual fidelity. So if we use standard approaches to regularization, we need to carefully chose out basies to recover the nice conceptual features of the smoothing spline measures.

In practice, it is not too difficult to do this, we insist on using an orthonormal basis for any space of basis transformations. This takes is the natural generalization of the scaling issue to a space of basis transformations, and solves the same problem. But this is not the easiest thing to explain, and is an easy issue to ignore.

And is there a similar scaling issue with interactions?

A similar issue exists with interactions.

For example, suppose we want to estimate a different slope for each level of a binary (two-valued) feature. If we use standard regularization approaches, we will penalize the estimate of the two interaction terms equally as compared to all the other features. Conceptually this over-penalizes the interaction features, as their estimates only affect predictions for a subset of our observations, while non-interaction feature's estimates affect the predictions for all of the observations.

$\endgroup$
1
  • $\begingroup$ Thanks for the long and thoughtful answer. (I'm only interested in prediction, by the way.) To restate the conclusion: if we choose an orthonormal basis, then the regularization term $\lambda \alpha^t \Omega \alpha$ = $\lambda \alpha^t \alpha$ anyway. Is that right? I can imagine that this leads to an odd-looking basis if we start with restricted cubic splines. Is it more common to use something like wavelets? B-splines? $\endgroup$ Feb 9, 2018 at 16:51
0
$\begingroup$

I will try to answer your question as far as I understand it:

1-)There is no problem using multiple transformations of the same variable by changing the scaling parameter. Actually, this is one of the reasons why you would wanna use the lasso. You try a bunch of scale parameters and you find the "optimum" one using cross-validation. Glmnet package in R does a great job for that.

2-) As far as I know, the only way to find the scale parameter is cross-validation. I dont think there is some kind of analytical formula that gives the best scale parameter.

3-) What do you mean by interaction?

$\endgroup$
1
  • $\begingroup$ Not sure this addresses the question...it sounds like you might be talking about the penalty strength hyperparameter, whereas it sounds like the original question was about scaling the regressors themselves (i.e. as is commonly done to ensure that the penalty applies equally to each). $\endgroup$
    – user20160
    Feb 8, 2018 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.