# Regularized least squares with a black box predictive function

In regularized OLS, the regularization parameter is applied to the weights $$\arg \min_w ||y - f(w)|| + \lambda ||w||^2 \\ f(w) = wx + b$$

Does it change the optimization at all if it was instead applied to the predictive function $$\arg \min_w ||y - f(w)|| + \lambda ||f(w)||^2$$ In my mind, the two optimization problems are equivalent.

If the $$f(w)$$ function is a black box instead of the linear model, does it matter if you're using $$\lambda ||w||^2$$ vs. $$\lambda ||f(w)||^2$$? I don't think the answer changes from the above case where $$f(w)$$ is known.

Yes it does and this is definitely not equivalent.

In

$$\arg \min_w ||y - f(w)|| + \lambda ||w||^2$$

The $$||y - f(w)||$$ part is about minimizing the squared error between target variable $$y$$ and predictions $$f(x)$$, i.e. you ask for $$f(x)$$ that is as close as possible to $$y$$ as measured by squared error. By including $$\lambda ||w||^2$$ you ask also, that if possible, you want the weights $$w$$ to be small, as measured with to $$L_2$$ norm. Regularizing the weights may shrink all, or some of them, to some degree.

If alternatively you used $$\lambda ||f(x)||^2$$, you would ask the predictions $$f(x)$$ to be the smallest possible, so you would be forcing the model to underestimate $$f(x)$$.

• That makes sense. Do we always apply the regularization to the weights? I saw in a course slide from MIT (I think) where it appears they applied it to the predictive function. I’m not on my computer now but I’ll find it tomorrow. Jun 17, 2020 at 8:41
• @dd22205 hard to comment without knowing the context, but LASSO is about regularizing weights. There are many different forms of regularization, not always applied to weights (not all models have weights at first place!).
– Tim
Jun 17, 2020 at 8:44
• Both Lasso and Tikhonov are about regularizing weights right? Here’s the link: mit.edu/~9.520/spring07/Classes/rlsslides.pdf. Looks like on slide 4 they regularize the hypothesis function. Jun 17, 2020 at 8:47
• @dd22205 correct. But there are also cases of constrained optimization where you put constrains on the parameters, or the function, maybe the slides you are referring to discussed something like this, hard to comment.
– Tim
Jun 17, 2020 at 8:51
• Another question I have is none of the values in the objective function are normalized, so the magnitude of the norm doesn't really mean anything to me. e.g., what if $y$ and $f(w)$ are on the order of billions, and the weights are say, on the order of thousands? In this case, how does regularization play a role? I think my inclination on this is that if you use cross-validation, it might show that you need a large $\lambda$ so that the first and second norms are comparable. For some reason I had thought $\lambda$ was bounded by [0, 1]. Jun 17, 2020 at 22:05