# Taylor approximation to penalty function

I'm reading through Fan's SCAD paper and I feel like I'm not getting a simple step. On page 1354 where he is talking about quadratic approximations to a penalty function he has

$$\left[\rho_\lambda(|\beta_j|)\right]' = \rho'_\lambda(|\beta_j|)\text{sgn}(\beta_j) \approx \left\{\rho'_\lambda(|\beta_{j0}|)/\beta_{j0}\right\}\beta_j$$

which I am comfortable with, but then he says $$\rho_\lambda(|\beta_j|) \approx \rho_\lambda(|\beta_{j0}|) + \dfrac{1}{2}\left\{\rho'_\lambda(|\beta_{j0}|)/\beta_{j0}\right\}(\beta_j^2-\beta_{j0}^2)$$ for $\beta_j \approx \beta_{j0}$. This looks pretty close to a 1st order taylor expansion about $\beta_{j0}$, but I'm not quite sure how the usual quadratic term (e.g. $(\beta_j - \beta_{j0})^2$) becomes the term above.

• It looks like a second order Taylor expansion to me. Apr 23 '14 at 9:01
• @Glen_b Hmmm, the (1/2) might be a mistake. The 1st-derivative has a $\beta_j$ at the end, so evaluated at $|\beta_{j0}|$ we get $|\beta_{j0}|\cdot (\beta_j -|\beta_{j0}|)$ and then they use $\beta_j \approx \beta_{j0}$ to get the difference of squares. But then again, why $|\beta_{j0}|\cdot \beta_j = \beta_j^2$? Where did the sign go? Apr 23 '14 at 9:05
• @Alecos It's no mistake: it comes from $\beta_j \approx \frac{1}{2}(\beta_j + \beta_{j0}),$ whence $\beta_{j0}(\beta_j-\beta_{j0}) \approx \frac{1}{2}(\beta_j + \beta_{j0})(\beta_j-\beta_{j0}) = \frac{1}{2}(\beta_j^2 - \beta_{j0}^2).$
– whuber
Apr 23 '14 at 14:20
• @whuber Yes, indeed. The $\approx$ sign can work miracles! Apr 23 '14 at 14:41
• Ah, thank you kind sirs. That is indeed some fine trickery there. Apr 23 '14 at 14:52

\begin{align*} \rho_\lambda(|\beta_j|) &\approx \rho_\lambda(|\beta_{j0}|) + \left.\rho_\lambda(|\beta_j|)'\right|_{\beta_{j0}}(\beta_j-\beta_{j0}) \quad\text{By Taylor expansion}\\ &\approx \rho_\lambda(|\beta_{j0}|) + \left\{\rho_\lambda'(|\beta_{j0}|)/\beta_{j0}\right\}\beta_{j0}(\beta_j-\beta_{j0}) \quad\text{By above approximation}\\ &\approx \rho_\lambda(|\beta_{j0}|) + \tfrac{1}{2}\left\{\rho_\lambda'(|\beta_{j0}|)/\beta_{j0}\right\}(\beta_j^2-\beta_{j0}^2) \end{align*}
$\text{Where the last approximation holds because$\beta_j \approx \beta_{j0}$} \implies \beta_{j0} \approx \tfrac{1}{2}(\beta_j+\beta_{j0})\implies \beta_{j0}(\beta_j-\beta_{j0}) \approx \tfrac{1}{2}(\beta_j+\beta_{j0})(\beta_j-\beta_{j0}) = \tfrac{1}{2}(\beta_j^2-\beta_{j0}^2)$