is LASSO's path deterministic? if yes, it is decided by what? if not, can we control it somehow? I am working on LASSO and I don't fully understand how the path works:
Where does the path start? does it start from the feature that gives the biggest variance of prediction?
How does LASSO select which feature to add next? 
Is the path (robustly) deterministic? By robust I mean, when I use a random subset of my data, the path should be fairly similar --- if not identical. If the path is not robust, then LASSO's outcome would be quite random, correct? Then we have to rely heavily on CV to make sure that the model is really predictive.
 A: LASSO is deterministic when applied to any single data sample from a population but it may select different predictors when applied to a different sample from the same population.
As shown in the link provided by @user18764 in a comment, LASSO trades off the mean-square prediction error against a penalty proportional to the sum of the absolute values of the regression coefficients. So, as the penalty on the coefficients is relaxed from infinity, LASSO necessarily starts with the feature that has the strongest single-variable relation to outcome. As the penalty is further relaxed the feature that makes the next best contribution is added. Features entered at early stages might be removed later in some cases, but the path to any final chosen penalty level is deterministic on that data sample.
The confusion enters when you take another sample from the underlying population and the predictive features are correlated. Then there is no assurance that the features most closely related to outcome in the new sample will be the same as those in the first sample. You might have similar predictive value in a model built on the second sample, but the specific predictors chosen could be different.
One good way to evaluate this issue is to repeat the entire LASSO model building process (including choice of penalty) on multiple bootstrap samples from your data and see (1) the variability of feature choices among the multiple models and (2) the predictive value of the multiple models when applied to the original complete data set. It's quite possible to have variability in feature choices and still have reasonably good predictive value.
