Bias and variance properties of $L^1$ vs $L^2$ normalization When moving from $L^2$ to $L^1$ normalization in Linear Regression, should I expect to see more bias or more variance? Note that bias is a sign of under fitting and variance is a sign of over fitting. Assume constant λ. 
I'm looking for a general rule of thumb here. If there isn't one, and the answer depends on some other factors I haven't accounted for, please explain. 
 A: The variance will increase, the problem with L1 regularisation is some of the coefficients are highly unpredictable.
The answer depends on how severe is your regularisation($\lambda$ value).
I generated sin wave with gaussian noise with different seeds and what is observed is as lower $\lambda$ value the l1 norm has a lower variance as $\lambda$ values increases the l2 norm has lower variance.

As you can see in the image, the alpha is the regularization coefficient. at $\alpha = 1e-15$ the corresponding we see the large difference between Rigde regression coefficients but, as alpha increases the deviation in ridge decreases drastically.
but when $\alpha = 10$ the lasso and ridge both have lower variance.
So as the alpha decreases the variance of ridge increases drastically.
PS: I have experimented with this setup multiple times and the trend is consistent.
So answering your question the variance will increase when you switch from L1 to L2 regularizer(and the scale of increase depends on $\lambda$ value.)

also i have added the dot products of $W$ vector.just to see how diffrent the $w$.
the diagonal entries are -ve which says theres a lot of diffrence between the $Ws$.
this will be further extended with more detailed anaylsis.
