3
$\begingroup$

I'm reading through Fan's SCAD paper and I feel like I'm not getting a simple step. On page 1354 where he is talking about quadratic approximations to a penalty function he has

$$\left[\rho_\lambda(|\beta_j|)\right]' = \rho'_\lambda(|\beta_j|)\text{sgn}(\beta_j) \approx \left\{\rho'_\lambda(|\beta_{j0}|)/\beta_{j0}\right\}\beta_j$$

which I am comfortable with, but then he says $$\rho_\lambda(|\beta_j|) \approx \rho_\lambda(|\beta_{j0}|) + \dfrac{1}{2}\left\{\rho'_\lambda(|\beta_{j0}|)/\beta_{j0}\right\}(\beta_j^2-\beta_{j0}^2)$$ for $\beta_j \approx \beta_{j0}$. This looks pretty close to a 1st order taylor expansion about $\beta_{j0}$, but I'm not quite sure how the usual quadratic term (e.g. $(\beta_j - \beta_{j0})^2$) becomes the term above.

Improve this question
$\endgroup$
5
  • $\begingroup$ It looks like a second order Taylor expansion to me. $\endgroup$
    – Glen_b
    Commented Apr 23, 2014 at 9:01
  • $\begingroup$ @Glen_b Hmmm, the (1/2) might be a mistake. The 1st-derivative has a $\beta_j$ at the end, so evaluated at $|\beta_{j0}|$ we get $|\beta_{j0}|\cdot (\beta_j -|\beta_{j0}|)$ and then they use $\beta_j \approx \beta_{j0}$ to get the difference of squares. But then again, why $|\beta_{j0}|\cdot \beta_j = \beta_j^2$? Where did the sign go? $\endgroup$
    – Alecos Papadopoulos
    Commented Apr 23, 2014 at 9:05
  • $\begingroup$ @Alecos It's no mistake: it comes from $\beta_j \approx \frac{1}{2}(\beta_j + \beta_{j0}),$ whence $\beta_{j0}(\beta_j-\beta_{j0}) \approx \frac{1}{2}(\beta_j + \beta_{j0})(\beta_j-\beta_{j0}) = \frac{1}{2}(\beta_j^2 - \beta_{j0}^2).$ $\endgroup$
    – whuber ♦
    Commented Apr 23, 2014 at 14:20
  • 1
    $\begingroup$ @whuber Yes, indeed. The $\approx$ sign can work miracles! $\endgroup$
    – Alecos Papadopoulos
    Commented Apr 23, 2014 at 14:41
  • $\begingroup$ Ah, thank you kind sirs. That is indeed some fine trickery there. $\endgroup$
    – bdeonovic
    Commented Apr 23, 2014 at 14:52

1 Answer 1

Reset to default
3
$\begingroup$

As @Alecos_papadopoulos and @whuber pointed out in the comments:

$ \begin{align*} \rho_\lambda(|\beta_j|) &\approx \rho_\lambda(|\beta_{j0}|) + \left.\rho_\lambda(|\beta_j|)'\right|_{\beta_{j0}}(\beta_j-\beta_{j0}) \quad\text{By Taylor expansion}\\ &\approx \rho_\lambda(|\beta_{j0}|) + \left\{\rho_\lambda'(|\beta_{j0}|)/\beta_{j0}\right\}\beta_{j0}(\beta_j-\beta_{j0}) \quad\text{By above approximation}\\ &\approx \rho_\lambda(|\beta_{j0}|) + \tfrac{1}{2}\left\{\rho_\lambda'(|\beta_{j0}|)/\beta_{j0}\right\}(\beta_j^2-\beta_{j0}^2) \end{align*}$

$\text{Where the last approximation holds because $\beta_j \approx \beta_{j0}$} \implies \beta_{j0} \approx \tfrac{1}{2}(\beta_j+\beta_{j0})\implies \beta_{j0}(\beta_j-\beta_{j0}) \approx \tfrac{1}{2}(\beta_j+\beta_{j0})(\beta_j-\beta_{j0}) = \tfrac{1}{2}(\beta_j^2-\beta_{j0}^2)$

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.