I'm reading through Fan's SCAD paper and I feel like I'm not getting a simple step. On page 1354 where he is talking about quadratic approximations to a penalty function he has

$$\left[\rho_\lambda(|\beta_j|)\right]' = \rho'_\lambda(|\beta_j|)\text{sgn}(\beta_j) \approx \left\{\rho'_\lambda(|\beta_{j0}|)/\beta_{j0}\right\}\beta_j$$

which I am comfortable with, but then he says $$\rho_\lambda(|\beta_j|) \approx \rho_\lambda(|\beta_{j0}|) + \dfrac{1}{2}\left\{\rho'_\lambda(|\beta_{j0}|)/\beta_{j0}\right\}(\beta_j^2-\beta_{j0}^2)$$ for $\beta_j \approx \beta_{j0}$. This looks pretty close to a 1st order taylor expansion about $\beta_{j0}$, but I'm not quite sure how the usual quadratic term (e.g. $(\beta_j - \beta_{j0})^2$) becomes the term above.

  • $\begingroup$ It looks like a second order Taylor expansion to me. $\endgroup$
    – Glen_b
    Apr 23, 2014 at 9:01
  • $\begingroup$ @Glen_b Hmmm, the (1/2) might be a mistake. The 1st-derivative has a $\beta_j$ at the end, so evaluated at $|\beta_{j0}|$ we get $|\beta_{j0}|\cdot (\beta_j -|\beta_{j0}|)$ and then they use $\beta_j \approx \beta_{j0}$ to get the difference of squares. But then again, why $|\beta_{j0}|\cdot \beta_j = \beta_j^2$? Where did the sign go? $\endgroup$ Apr 23, 2014 at 9:05
  • $\begingroup$ @Alecos It's no mistake: it comes from $\beta_j \approx \frac{1}{2}(\beta_j + \beta_{j0}),$ whence $\beta_{j0}(\beta_j-\beta_{j0}) \approx \frac{1}{2}(\beta_j + \beta_{j0})(\beta_j-\beta_{j0}) = \frac{1}{2}(\beta_j^2 - \beta_{j0}^2).$ $\endgroup$
    – whuber
    Apr 23, 2014 at 14:20
  • 1
    $\begingroup$ @whuber Yes, indeed. The $\approx$ sign can work miracles! $\endgroup$ Apr 23, 2014 at 14:41
  • $\begingroup$ Ah, thank you kind sirs. That is indeed some fine trickery there. $\endgroup$
    – bdeonovic
    Apr 23, 2014 at 14:52

1 Answer 1


As @Alecos_papadopoulos and @whuber pointed out in the comments:

$ \begin{align*} \rho_\lambda(|\beta_j|) &\approx \rho_\lambda(|\beta_{j0}|) + \left.\rho_\lambda(|\beta_j|)'\right|_{\beta_{j0}}(\beta_j-\beta_{j0}) \quad\text{By Taylor expansion}\\ &\approx \rho_\lambda(|\beta_{j0}|) + \left\{\rho_\lambda'(|\beta_{j0}|)/\beta_{j0}\right\}\beta_{j0}(\beta_j-\beta_{j0}) \quad\text{By above approximation}\\ &\approx \rho_\lambda(|\beta_{j0}|) + \tfrac{1}{2}\left\{\rho_\lambda'(|\beta_{j0}|)/\beta_{j0}\right\}(\beta_j^2-\beta_{j0}^2) \end{align*}$

$\text{Where the last approximation holds because $\beta_j \approx \beta_{j0}$} \implies \beta_{j0} \approx \tfrac{1}{2}(\beta_j+\beta_{j0})\implies \beta_{j0}(\beta_j-\beta_{j0}) \approx \tfrac{1}{2}(\beta_j+\beta_{j0})(\beta_j-\beta_{j0}) = \tfrac{1}{2}(\beta_j^2-\beta_{j0}^2)$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.