Why is selection of one amongst the group of highly correlated variables by lasso is stated as its disadvantage? I have seen PCA being used to eliminate variables for dimensionality reduction, but LASSO's ability to eliminate variables for the very same purpose is often described as a disadvantage.
My intuition is that in PCA we still retain some of the information in the reduced dimensions as the newly formed features are linear combinations of the old variables, whereas in LASSO we just completely drop some of the original variables by setting the coefficients of non selected features to zero. Is my intuition correct? If not, then why is dimensionality reduction a drawback in one scenario and an advantage in the other?
 A: Principal Components Analysis (PCA) doesn't select among the original features unless the features are orthogonal. It lowers the dimension of the feature space by ignoring PCs (linear combinations of potentially all the predictors) that explain little of the feature-set variance, but in general it does not remove any of the original features from the problem.
Some find the ability of LASSO to choose among several correlated features to be a strength rather than a weakness, particularly when the number of features greatly exceeds the number of cases. Yes, LASSO's choice among correlated features can be highly sample dependent. Nevertheless, as LASSO penalizes the coefficients for the retained variables it can still be useful for developing prediction models. The "disadvantage" is when one tries to interpret the selection by LASSO in terms of "variable importance" for inference.
The issues raised in this question point out a potential advantage of ridge regression, which is effectively PCA with varying weights among the PCs rather than simple 0/1 weights. (See page 79 of ESLII.) Try both LASSO and ridge regression on multiple bootstrap samples of a data set having correlated predictors. The particular features selected by LASSO may change dramatically among the bootstrap samples. This illustrates the mistake of interpreting choices by LASSO in terms of an underlying true "variable importance." The coefficients of correlated predictors in ridge regression may be variable among bootstrap samples but the ridge models will contain all the features. 
A: Using PCA to do feature selection is different than using LASSO because PCA does not consider the response variable, but only check the variance in features.
See my answers here, with graphical illustration (logistic regression case / linear method without regularization).
How to decide between PCA and logistic regression?
