# Regularization hyperparameter tuning without cross validation

Given a model on the observation data vector $$y$$ at coordinate vector $$x$$ $$M(f,\lambda) := (1-\lambda)L(y,f(x))+\lambda J(f)$$ where $$\lambda\in[0,1]$$, $$L$$ is a loss function and $$J$$ a roughness function and $$f$$ is a function of $$x$$. I perform $$f_\lambda=\text{argmin}_f \,M(f,\lambda)$$

The standard method for tuning $$\lambda$$ is cross validation. In order to save on computational cost, is it completely nuts to minimize some function over $$\lambda$$, say $$\min_\lambda\, N(\lambda,a)$$ where $$N(\lambda,a):=(1-a)\ln L(y,f_\lambda(x)) +a\ln J(f_\lambda)$$ for some chosen $$a\in[0,1]$$? Are there other related approaches for cheaply tuning the regularization parameter?

I suppose in principle for any given $$\lambda_c$$ obtained from cross validation there is always an $$a$$ such that $$\lambda_c$$ is the value $$\lambda$$ takes to minimize $$N(\lambda,a)$$ for that particular $$a$$. So I see there is much ambiguity in this question. But I am hoping to hear some empirical experience of obtaining directly a $$N$$ function.

• It looks like in your $N(\lambda)$ the $a$ plays the role that $\lambda$ had before - it even looks like finding the optimal $a$ could amount to finding the optimal $\lambda$, in which case the $N(\lambda)$ isn't going to help you to find $\lambda$... people do cross-validation for a reason... Commented Nov 22, 2020 at 21:10
• @Lewian: Are you saying the same thing as the first sentence of my last paragraph?
– Hans
Commented Nov 22, 2020 at 21:28
• I don't quite understand that sentence, so I don't know. Commented Nov 22, 2020 at 22:14
• @Lewian: I edited that sentence. Is it clear to you now?
– Hans
Commented Nov 23, 2020 at 1:15
• Maybe, but certainly this is not what I was saying. You make a mathematical statement which may be true or false, I don't know. What I was saying was mainly that introducing a new parameter $a$ of which it is not clear how to choose it doesn't seem to make the problem of finding another parameter $\lambda$ any easier. And on top of that that the role played by the $a$ seems to be very similar to the role played by $\lambda$ before, so it looks like you're only pushing the problem around rather than getting closer to solving it. Commented Nov 23, 2020 at 9:14

$$N(\lambda, a)$$ in general is not a good function for tuning the regularization parameter $$\lambda$$ without more constraint. In general, $$L(y,f_L(x))=0$$ for $$f_L(x)=y$$ and $$J(f)=0$$ for some $$f=f_J$$. Then $$\ln(L(y,f_L))=\ln(J(f_J))=-\infty$$. Obviously $$f_{\lambda=0}=f_L\,\bigvee\, f_{\lambda=1}=f_J\implies N(\lambda,a)=-\infty$$. Thus there is no regularization at all.

A good candidate for $$N(\lambda,a)$$ may be $$N(\lambda,a):=(1-a)\ln\big(1+L(y,f_\lambda(x))\big)+a\ln \big(1+J(f_\lambda)\big)$$ avoiding $$-\infty$$ as a function value.