Why would LASSO not shrink irrelevant features to zero? Assume I have 10 features to predict an outcome and I use LASSO regression. Let's say the RMSE of the test set is 20.
Now, I introduce 5 more features and predict the same outcome, and I also use LASSO. The RMSE of the test set is getting larger, to 80. 
Would it be possible? Why would it happen? 
LASSO shrinks the parameters to 0. And if the new features could not improve the prediction ability, why would it not shrink the new features to 0 making the new model become the original model? In terms of this, wouldn't it produce a similar RMSE for the test set, not a very different one?
 A: It's true that LASSO encourages sparseness in the sense that $\beta$s which are very close to zero are set to zero exactly. So if the tendency is to maintain features which have large values of $\beta$ this may not lead to better prediction in the sense of RMSE. 
A $\beta$ for a newly introduced feature may be very large because that feature has a low prevalence or low variability, so it enhances prediction in a small group of observations that is very different at the sacrifice of loosing predictive accuracy among the masses who were discriminated better by smaller $\beta$s
As an example here is a trivariate relationship between a continuous feature $x$ and a binary feature $w$ with an outcome $Y$. The $x$ effect explains much more of the variability in these data than the $w$ effect despite the $x$ effect being smaller overall than the $w$ effect, even after standardization. Lasso would favor $w$ in a model over $x$ because of its magnitude, but that alone does not suffice to lead to good prediction, we merely select $w$ because it is good at discriminating participants. This is the type of feature LASSO tends to select.

This underscores the importance of using cross-validation to select the tuning parameter in a LASSO model. In one such as this, you would find a much better predictive accuracy by including both effects.
A: The straightforward answer is that LASSO does not always work as intended.  Given its penalty factor LASSO has no way of differentiating between a true causal variable that has a high coefficient and should be selected in your model vs. another variable that has little relationship with Y and has a low coefficient. The LASSO algorithm may randomly and often select the weak variable instead of the strong causal variable. And, that's a big problem.  In doing so, the LASSO model not only dismantles the explanatory logic of your original model; it, also typically crashes into making much poorer predictions than your original model. 
You can visualize representations of such problems by doing searches for Images and search specifically "LASSO coefficient path" and "LASSO MSE graph."  The first set of graphs will show how often LASSO will choose weak variables instead of strong causal ones.  The second graph (MSE) will show how numerous LASSO models do a poor job at prediction whereby the best model is actually the original one associated with a penalty factor of zero. 
