Frequentist view 👀
In one sense, we can think of both regularizations as "shrinking the weights"; L2 minimizes the Euclidean norm of the weights, while L1 minimizes the Manhattan norm. Following this line of thinking, we can reason that the equipotentials of L1 and L2 are spherical and diamond-shaped respectively, so L1 is more likely to lead to sparse solutions, as illustrated in Bishop's Pattern Recognition and Machine Learning:
Bayesian view 👀
However, in order to understand how priors relate to the linear model, we need to understand the Bayesian interpretation of ordinary linear regression. Katherine Bailey's blogpost is an excellent read for this. In a nutshell, we assume normally-distributed i.i.d. errors in our linear model
$$\mathbf{y} = \mathbf{\theta}^\top\mathbf{X} + \mathbf\epsilon$$
i.e. each of our $N$ measurements $y_i, i = 1, 2, \ldots, N$ have a noise $\epsilon_k\sim \mathcal{N}(0,\sigma)$.
Then we can say that our linear model has a Gaussian likelihood too! The likelihood of $\mathbf{y}$ is
\begin{equation}
p(\mathbf{y}|\mathbf{X}, \mathbf{\theta}; \mathbf{\epsilon}) = \mathcal{N}(\mathbf{\theta}^\top\mathbf{X}, \mathbf{\sigma})
\end{equation}
As it turns out... The maximum likelihood estimator is identical to minimizing the squared error between predicted and actual output values under the normality assumption for the error.
\begin{align*}
{\bf \hat{\theta}_{\text{MLE}}} &= \arg\max_{\bf \theta} \log P(y | \theta) \\
&=\underset{\theta}{\arg\min} \sum_{i=1}^n(y_i - \theta^\top{\mathbf{x}_i})^2
\end{align*}
Regularization as putting priors on weights
If we were to place a non-uniform prior $P(\theta)$ on the weights of linear regression, the maximum a posteriori probability (MAP) estimate would be:
\begin{equation*}
{\bf \hat{\theta}_{\text{MAP}}} = \arg\max_{\bf \theta} \log P(y | \theta) + \log P(\theta)
\end{equation*}
As derived in Brian Keng's blogpost, if $P(\theta)$ is a Laplace distribution it's equivalent to L1 regularization on $\theta$.
Similarly, if $P(\theta)$ is a Gaussian distribution, it's equivalent to L2 regularization on $\theta$.
Now we have another view into why putting a Laplace prior on the weights is more likely to induce sparsity: because the Laplace distribution is more concentrated around zero, our weights are more likely to be zero.
