Given a model on the observation data vector $y$ at coordinate vector $x$ $$M(f,\lambda) := (1-\lambda)L(y,f(x))+\lambda J(f) $$ where $\lambda\in[0,1]$, $L$ is a loss function and $J$ a roughness function and $f$ is a function of $x$. I perform $$f_\lambda=\text{argmin}_f \,M(f,\lambda)$$

The standard method for tuning $\lambda$ is cross validation. In order to save on computational cost, is it completely nuts to minimize some function over $\lambda$, say $$\min_\lambda\, N(\lambda,a)$$ where $$N(\lambda,a):=(1-a)\ln L(y,f_\lambda(x)) +a\ln J(f_\lambda)$$ for some chosen $a\in[0,1]$? Are there other related approaches for cheaply tuning the regularization parameter?

I suppose in principle for any given $\lambda_c$ obtained from cross validation there is always an $a$ such that $\lambda_c$ is the value $\lambda$ takes to minimize $N(\lambda,a)$ for that particular $a$. So I see there is much ambiguity in this question. But I am hoping to hear some empirical experience of obtaining directly a $N$ function.

  • $\begingroup$ It looks like in your $N(\lambda)$ the $a$ plays the role that $\lambda$ had before - it even looks like finding the optimal $a$ could amount to finding the optimal $\lambda$, in which case the $N(\lambda)$ isn't going to help you to find $\lambda$... people do cross-validation for a reason... $\endgroup$ Nov 22, 2020 at 21:10
  • $\begingroup$ @Lewian: Are you saying the same thing as the first sentence of my last paragraph? $\endgroup$
    – Hans
    Nov 22, 2020 at 21:28
  • $\begingroup$ I don't quite understand that sentence, so I don't know. $\endgroup$ Nov 22, 2020 at 22:14
  • $\begingroup$ @Lewian: I edited that sentence. Is it clear to you now? $\endgroup$
    – Hans
    Nov 23, 2020 at 1:15
  • 2
    $\begingroup$ Maybe, but certainly this is not what I was saying. You make a mathematical statement which may be true or false, I don't know. What I was saying was mainly that introducing a new parameter $a$ of which it is not clear how to choose it doesn't seem to make the problem of finding another parameter $\lambda$ any easier. And on top of that that the role played by the $a$ seems to be very similar to the role played by $\lambda$ before, so it looks like you're only pushing the problem around rather than getting closer to solving it. $\endgroup$ Nov 23, 2020 at 9:14

1 Answer 1


$N(\lambda, a)$ in general is not a good function for tuning the regularization parameter $\lambda$ without more constraint. In general, $L(y,f_L(x))=0$ for $f_L(x)=y$ and $J(f)=0$ for some $f=f_J$. Then $\ln(L(y,f_L))=\ln(J(f_J))=-\infty$. Obviously $f_{\lambda=0}=f_L\,\bigvee\, f_{\lambda=1}=f_J\implies N(\lambda,a)=-\infty$. Thus there is no regularization at all.

A good candidate for $N(\lambda,a)$ may be $$N(\lambda,a):=(1-a)\ln\big(1+L(y,f_\lambda(x))\big)+a\ln \big(1+J(f_\lambda)\big)$$ avoiding $-\infty$ as a function value.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.