Regularization hyperparameter tuning without cross validation

Question

Given a model on the observation data vector $y$ at coordinate vector $x$ $$M(f,\lambda) := (1-\lambda)L(y,f(x))+\lambda J(f) $$ where $\lambda\in[0,1]$, $L$ is a loss function and $J$ a roughness function and $f$ is a function of $x$. I perform $$f_\lambda=\text{argmin}_f \,M(f,\lambda)$$

The standard method for tuning $\lambda$ is cross validation. In order to save on computational cost, is it completely nuts to minimize some function over $\lambda$, say $$\min_\lambda\, N(\lambda,a)$$ where $$N(\lambda,a):=(1-a)\ln L(y,f_\lambda(x)) +a\ln J(f_\lambda)$$ for some chosen $a\in[0,1]$? Are there other related approaches for cheaply tuning the regularization parameter?

I suppose in principle for any given $\lambda_c$ obtained from cross validation there is always an $a$ such that $\lambda_c$ is the value $\lambda$ takes to minimize $N(\lambda,a)$ for that particular $a$. So I see there is much ambiguity in this question. But I am hoping to hear some empirical experience of obtaining directly a $N$ function.

Answer 1

$N(\lambda, a)$ in general is not a good function for tuning the regularization parameter $\lambda$ without more constraint. In general, $L(y,f_L(x))=0$ for $f_L(x)=y$ and $J(f)=0$ for some $f=f_J$. Then $\ln(L(y,f_L))=\ln(J(f_J))=-\infty$. Obviously $f_{\lambda=0}=f_L\,\bigvee\, f_{\lambda=1}=f_J\implies N(\lambda,a)=-\infty$. Thus there is no regularization at all.

A good candidate for $N(\lambda,a)$ may be $$N(\lambda,a):=(1-a)\ln\big(1+L(y,f_\lambda(x))\big)+a\ln \big(1+J(f_\lambda)\big)$$ avoiding $-\infty$ as a function value.