I'm confused about the way L1 & L2 pop-up in what seem different roles in the same play:
Are [1] and [2] the same thing? or are these two completely separate practices sharing the same names? (if relevant) what are the similarities and differences between the two?
They are distinct notions.
Point #2 refers to the usual kind of loss function. The first example almost anyone who studies statistics or data analysis of any kind sees is the square loss in ordinary least squares linear regression: add up the squared residuals.
$$ L(y,\hat y)=\overset{n}{\underset{i=1}{\sum}}\left( y_i - \hat y_i \right)^2 $$
Another viable loss function is to add up the absolute residuals.
$$ L(y,\hat y)=\overset{n}{\underset{i=1}{\sum}}\left\vert y_i - \hat y_i \right\vert $$
Each of these can be expressed in terms of $p$-norms.
$$ L(y,\hat y)=\overset{n}{\underset{i=1}{\sum}}\left( y_i - \hat y_i \right)^2=\vert\vert y-\hat y\vert\vert_2^2\\ L(y,\hat y)=\overset{n}{\underset{i=1}{\sum}}\left\vert y_i - \hat y_i \right\vert = \vert\vert y-\hat y\vert\vert_1 $$
Consequently, it is reasonable to refer to these as $\ell_2$ and $\ell_1$ loss, respectively.
The penalization from the regularization in point #1 is separate. First, there is not necessarily a need to include penalization, so it might be that you just find the regression parameters that lead to predictions $\hat y_i$ giving the minimal $\ell_p$ loss, and this is exactly what ordinary least squares estimation does for $\ell_2$ loss. However, there are various reasons why the best loss value might not be desirable. Regularization is a way of sacrificing the training loss value in order to improve some other facet of performance, a major example being to sacrifice the in-sample fit of a machine learning model to quell overfitting and improve out-of-sample performance.
You can mix-and-match loss functions and regularization to your heart's content. For instance, ridge regression uses square loss with an added penalty term that involves the $\ell_2$ norm of the regression parameter vector.
$$ L_{\text{ridge}}=\vert\vert y-\hat y\vert\vert^2_2 + \lambda\vert\vert\hat\beta\vert\vert_2 $$
LASSO regression uses square loss with a penalty term that uses the $\ell_1$ norm of the parameter vector.
$$ L_{\text{LASSO}}=\vert\vert y-\hat y\vert\vert^2_2 + \lambda\vert\vert\hat\beta\vert\vert_1 $$
Elastic net uses both types of penalty.
$$ L_{\text{Elastic Net}}=\vert\vert y-\hat y\vert\vert^2_2 + \lambda_1\vert\vert\hat\beta\vert\vert_1+ \lambda_2\vert\vert\hat\beta\vert\vert_2 $$
Finally, while I do not see this approach discussed much, you could use $\ell_1$ loss with either penalty or even both.
$$ L_{\text{Other}}=\vert\vert y-\hat y\vert\vert_1 + \lambda_1\vert\vert\hat\beta\vert\vert_1+ \lambda_2\vert\vert\hat\beta\vert\vert_2 $$
(The $\lambda$ parameters control how much of a penalty there is for having large coefficients in the parameter vector. It is common to tune these using cross validation.)
Getting to other types of models, nothing stops you from using $\ell_1$ or $\ell_2$ penalization (or both) with, say, logistic regression and its associated "log loss".
$$ L_{\text{Log}}=-\overset{n}{\underset{i=1}{\sum}}\left( y_i\log(\hat y_1) + (1 - y_1)\log(1 - \hat y_1) \right)\\ L_{\text{Penalized Log}}=-\overset{n}{\underset{i=1}{\sum}}\left( y_i\log(\hat y_1) + (1 - y_1)\log(1 - \hat y_1) \right)+ \lambda_1\vert\vert\hat\beta\vert\vert_1+ \lambda_2\vert\vert\hat\beta\vert\vert_2 $$
\ell_2 in $\LaTeX$) coincides with maximum likelihood estimation for $iid$ Gaussian errors. This allows for many properties to be derived analytically, which was great when computing power was more limited. That leads to a less-good reason, which is tradition.
$\endgroup$
no, they refer to two different things:
MAE and L1 is a Laplacian prior, MSE and L2 is a Gaussian likelihood/prior, and they are both derived using the max log-likelihood principle
The terms "L1" and "L2" refer to special functions called norms, which measure the length or size of a vector. You are correct in that they are used in two different contexts in statistics and machine learning, but their meanings are the same in both contexts.
In the context of regularization, the L1 and/or L2 norm restricts the magnitude of the parameter vector of a model. The difference between L1 and L2 regularization comes down to the differences between the L1 and L2 norms. See e.g. https://medium.com/analytics-vidhya/effects-of-l1-and-l2-regularization-explained-5a916ecf4f06. As pointed out in other answers, L1 regularization in a regression model corresponds to a Laplace prior on coefficients in Bayesian modeling, and L2 regularization corresponds to a Gaussian prior.
In the context of loss functions, the L1 or L2 norm measures the magnitude of the error vector of the model on a train/test/validation set. L1 loss is the Median Absolute Error (MAE), and L2 loss is the Root Mean Squared Error (RMSE). As pointed out in the comments, regression models fitted with L1 loss are models of a conditional median, while models fitted with L2 loss are models of a conditional expectation (conditional mean). The latter also happens to correspond with a Gaussian GLM maximum-likelihood model, where the conditional distribution of the data follows a Gaussian distribution centered at the regression prediction.
The L2 norm corresponds to our conventional notion of Euclidean distance, which is essentially a multi-dimensional extension of the Pythagorean theorem. You can think of Euclidean distances as the lengths of hypotenuses of right triangles drawn between points.
The L1 norm corresponds to the weirder notion of Manhattan (aka "Taxicab") distance, so named because distances resemble the distance traveled by a taxi cab following the grid layout of streets in Manhattan, New York.
It's very common in statistics and machine learning to use L2 loss (MSE) with L1 regularization, or even both L1 and L2 regularization in the same model.
L1 loss (MAE) is much less common than L2 in general, in part because the absolute value is not differentiable. However there is a "smooth" differentiable L1 loss that attempts to mimic the properties of true L1 loss, see e.g. How to interpret smooth l1 loss?.
