4
$\begingroup$

I am wondering how I can visualize or understand the $L^{0.5}$ norm in regression settings. In other words, the loss function is

$$ \sum_{i=1}^{n}\left(Y_i-\sum_{j=1}^{p} X_{ij}\beta_j\right)^2 + \lambda \sum_{j=1}^p \beta_j^{0.5} $$

Are there any tricks to seeing what it looks like? Thanks.

Update: How could I perhaps plot it to look like the contours below, which are for L1 and L2 regularization (from Hastie and Tibshirani, Elements of Statistical Learning)?

enter image description here

$\endgroup$
8
  • 1
    $\begingroup$ Concerning your update: See figure 3.12 on page 72 of the ESL (Hastie & Tibshirani). $\endgroup$ Apr 26 '21 at 7:45
  • 1
    $\begingroup$ @Yves that is incorrect. You can refer to Wikipedia, to give one counterexample en.wikipedia.org/wiki/Lp_space#The_p-norm_in_finite_dimensions Moreover, using $p$ smaller than 1 is sometimes used in machine learning. $\endgroup$
    – Tim
    Apr 26 '21 at 10:47
  • 2
    $\begingroup$ The Wikipedia article seems to say that $L^{0.5}$ would not be a norm. $\endgroup$
    – Dave
    Apr 26 '21 at 11:01
  • 1
    $\begingroup$ @Yves It doesn't function as a norm per se in this application: it's just a formula for a penalty. Certainly the value exists: it can be computed provided the $\beta_j\ne 0$ (notice, though, that the formula requires absolute values around the $\beta_j$!). $\endgroup$
    – whuber
    Apr 26 '21 at 15:08
  • 1
    $\begingroup$ @whuber yes be the penaly (the power of ) a norm or not does not matter here. However the "$L^{0.5}$ norm" may offend mathematicians:) $\endgroup$
    – Yves
    Apr 26 '21 at 16:34
5
$\begingroup$

If you have only two $\beta_j$ parameters, just plot it in a 3D plot with $\beta_1$ on $x$-axis, $\beta_2$ on $z$-axis, and the loss on $y$-axis. If there is more parameters, there is no easy way to plot them. What you can do, it to use a dimensionality reduction algorithm to reduce the dimensionality of inputs, as authors of the loss landscape paper did, but in such case keep in mind that you will no longer plotting the function, but a transformation of it, that may, or may not reflect it well.

The 3.11 figure by Hastie et al shows just the geometry of the function. This can easily be done for two dimensions in any plotting software, you can find an example in Python's Matplotlib below.

import matplotlib.pyplot as plt
import numpy as np

def reg(x, p):
    return np.abs(x) ** p

x, y = np.meshgrid(np.linspace(-4, 4, num=100), np.linspace(-4, 4, num=100))
pars = [0.5, 1, 2]

plt.figure(figsize=(15,4))

for (i, p) in enumerate(pars, start=1):
    z = reg(x, p) + reg(y, p)
    plt.subplot(1, len(pars), i)
    plt.contourf(x,y,z)
    plt.title(r"$|\beta_1|^{%s} + |\beta_2|^{%s}$" % (p, p))
    
plt.show()

Plot showing three functions |beta1|^p + |beta2|^p for p in 0.5, 1, and 2.

$\endgroup$
2
  • $\begingroup$ Thank you, I forgot to put in a graph above I saw once of $L1$ and $L2$ regularization. Is it possible to do the plot for $L^{0.5}$? $\endgroup$
    – user321627
    Apr 26 '21 at 7:41
  • $\begingroup$ Thank you, this is amazing! $\endgroup$
    – user321627
    Apr 27 '21 at 7:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.