Skip to main content
added 194 characters in body
Source Link
Mark L. Stone
  • 13.5k
  • 1
  • 38
  • 58

$\|x\|_2^2$ and $\|x\|_1$ are $L_p^p$ for p = 2 and 1 respectively, so they are consistent in that sense.

Nevertheless, there's no really standard convention on whether or not an L2 regularization term should be squared. In fact, there's not even standardization on whether the "baseline" portion of a "least squares" objective should be squared. You could have { $norm(Ax-b)$ or $norm(Ax-b)^2$ } plus a regularization term. In the absence of the regularization term, it doesn't matter, but with a regularization term, it does. So (Squaring the norm is the "standard" way of doing it, e.g., least squares). So squaring or not the baseline portion of the objective and the regularization term (even if based on an $L_2$ norm) provides 4 different possible ways in effect of doing regularization, and although similar, they are all different.

And of course, we can change from the 2 norm to the 1 norm or something else in the baseline objective term $norm(Ax-b)$. The infinity norm would give a minimax solution, by minimizing the maximum absolute deviation of elements of $Ax-b$.

Bottom line: regularization tends to be somewhat of an ad hoc or arbitrary process, and there's more than one way to do it.

$\|x\|_2^2$ and $\|x\|_1$ are $L_p^p$ for p = 2 and 1 respectively, so they are consistent in that sense.

Nevertheless, there's no really standard convention on whether or not an L2 regularization term should be squared. In fact, there's not even standardization on whether the "baseline" portion of a "least squares" objective should be squared. You could have { $norm(Ax-b)$ or $norm(Ax-b)^2$ } plus a regularization term. In the absence of the regularization term, it doesn't matter, but with a regularization term, it does. So squaring or not the baseline portion of the objective and the regularization term (even if based on an $L_2$ norm) provides 4 different possible ways in effect of doing regularization, and although similar, they are all different.

And of course, we can change from the 2 norm to the 1 norm or something else in the baseline objective term $norm(Ax-b)$.

Bottom line: regularization tends to be somewhat of an ad hoc or arbitrary process, and there's more than one way to do it.

$\|x\|_2^2$ and $\|x\|_1$ are $L_p^p$ for p = 2 and 1 respectively, so they are consistent in that sense.

Nevertheless, there's no really standard convention on whether or not an L2 regularization term should be squared. In fact, there's not even standardization on whether the "baseline" portion of a "least squares" objective should be squared. You could have { $norm(Ax-b)$ or $norm(Ax-b)^2$ } plus a regularization term. In the absence of the regularization term, it doesn't matter, but with a regularization term, it does. (Squaring the norm is the "standard" way of doing it, e.g., least squares). So squaring or not the baseline portion of the objective and the regularization term (even if based on an $L_2$ norm) provides 4 different possible ways in effect of doing regularization, and although similar, they are all different.

And of course, we can change from the 2 norm to the 1 norm or something else in the baseline objective term $norm(Ax-b)$. The infinity norm would give a minimax solution, by minimizing the maximum absolute deviation of elements of $Ax-b$.

Bottom line: regularization tends to be somewhat of an ad hoc or arbitrary process, and there's more than one way to do it.

Source Link
Mark L. Stone
  • 13.5k
  • 1
  • 38
  • 58

$\|x\|_2^2$ and $\|x\|_1$ are $L_p^p$ for p = 2 and 1 respectively, so they are consistent in that sense.

Nevertheless, there's no really standard convention on whether or not an L2 regularization term should be squared. In fact, there's not even standardization on whether the "baseline" portion of a "least squares" objective should be squared. You could have { $norm(Ax-b)$ or $norm(Ax-b)^2$ } plus a regularization term. In the absence of the regularization term, it doesn't matter, but with a regularization term, it does. So squaring or not the baseline portion of the objective and the regularization term (even if based on an $L_2$ norm) provides 4 different possible ways in effect of doing regularization, and although similar, they are all different.

And of course, we can change from the 2 norm to the 1 norm or something else in the baseline objective term $norm(Ax-b)$.

Bottom line: regularization tends to be somewhat of an ad hoc or arbitrary process, and there's more than one way to do it.