I saw this notation for ordinary least squares here.

$$ \min_w \left\| Xw - y \right\|^2_2$$

I have never seen the double bars and the 2 at the bottom. What do these symbols mean? Do they have specific terminology for them?

  • 5
    $\begingroup$ The use of the double bars just indicates that we are using the L2 norm. $\endgroup$ Commented Jan 5, 2017 at 17:18
  • $\begingroup$ @MichaelChernick and the 2? Is that part of "L2 norm"? $\endgroup$ Commented Jan 5, 2017 at 17:19
  • 1
    $\begingroup$ Yes, like L2, there is also L1. $\endgroup$
    – Jon
    Commented Jan 5, 2017 at 17:22
  • $\begingroup$ I think $X_w$ should be $Xw$ since $w$ is a vector $\endgroup$
    – ilanman
    Commented Jan 5, 2017 at 18:01
  • $\begingroup$ @ilanman Yes that is what was in the notation before the edit. I changed it back $\endgroup$ Commented Jan 5, 2017 at 18:07

1 Answer 1


You're talking about the $\ell_2$-norm (Euclidean norm) of the vector ($Xw - y$). If this foreign to you, briefly, the $\ell_p$-norm of a vector $u \in \mathbb{R}^{n}$, is:

$$\|u\|_p = \big(\sum_{i=1}^{n} |u_i|^p\big)^{\frac1p} $$

So in your case $\|u\|_2^2 = (\big(\sum\limits_{i=1}^{n} |u_i|^2\big)^{\frac12})^2 = \sum\limits_{i=1}^{n} u_i^2$ which is consistent with the sum of squared residuals for a linear regression. In the context of regression problems, you'll also see this a lot in mean squared error (MSE) calculations, and in ridge regression.

This is a common norm (among other reasons, it's mathematically convenient), so when it's obvious from the context, you'll see the lower $2$ omitted, and just $\|u\|^2$.

As mentioned in the comments, you may also see the $\ell_1$-norm:

$$\|u\|_1 = \sum_{i=1}^{n} |u_i| $$

Which corresponds to the absolute value. Again, you'll see this in mean absolute error (MAE) or lasso problems.

Other popular norms:

  • $\ell_0$-norm: Hamming distance, or # of non-zeros in a vector, i.e. in calculating the sparsity of a vector. Technically this isn't a norm (it's a cardinality function), because you have a $\frac{1}{0}$ term in the definition, but it has the form of a norm so we call it one.
    • This norm is the ideal norm used in inducing sparsity for regression problems since we really want to zero out coefficients, however computing $\ell_0$ regularization is NP-hard, so instead we approximate it with $\ell_1$ which is solvable via linear programming. It's also popular in Compressed Sensing.
  • $\ell_{\infty}$-norm: = $\underset{i} {\text{max}}$ $\{|x_i|\}$ for $i = 1, ..., n$
  • $\|A\|_F$: Frobenius (Euclidean) norm, applied to a matrix $A \in \mathbb{R}^{n\times m} = \sqrt{\sum \limits_{i=1}^{n}\sum \limits_{j=1}^{m}|a_{ij}|^2}$
  • 2
    $\begingroup$ The link to wolfram alpha was really helpful. $\endgroup$ Commented Jan 5, 2017 at 18:12
  • $\begingroup$ You write that the $\ell_0$ (pseudo)norm counts the number of zeroes in a vector—did you perhaps mean the number of non-zero entries? (This would be more consistent with what I've seen, and also would mean that $\lVert u \rVert_{0}$ would be the Hamming distance between $u$ and $0 \in \mathbb R^n$, as opposed to being $n$ minus that distance.) $\endgroup$
    – wchargin
    Commented Jan 5, 2017 at 22:16
  • 1
    $\begingroup$ Spelling error: "Frobenius". $\endgroup$
    – hobbs
    Commented Jan 6, 2017 at 8:22
  • 1
    $\begingroup$ Instead of "this is a common norm" I would've just said "L2 is the norm" ;) $\endgroup$
    – user541686
    Commented Jan 6, 2017 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.