How to select the best feature set from Ridge regression? I have applied L2 regularization on my features and have got coefficient values as below(hiding the column name due to client work:

I am unsure about what all features should I choose? Should I select all the features except for one with value = 0, or is there any formula to choose the best features?
Appreciate your help.
 A: The beauty of ridge regression (L2 norm / quadratic penalty / Bayesian model with normal priors) is that it doesn't waste any predictive information trying to determine which variables have exactly zero effect.  Any attempt to select variables after the fact will result in worse predictions.  The set of "selected" variables will have a zero probability of being the truly predictive variable.
If you truly believe that some of the unknown coefficients are exactly zero (why?) you can formulate a better model by using Bayesian priors for regression coefficients that reflect your beliefs.  For example if you knew that 1/5th of the unknown coefficients are exactly zero but didn't know which 1/5th, you could form a prior that is normal (as with ridge) but is mixed with a spike at zero such that $\Pr(\beta = 0) = \frac{1}{5}$.  A great advantage of the Bayesian approach is that you automatically get exact inference when you're done whereas traditional inference after traditional feature selection is very hard to do.
Note that it is not valid to use simple methods after using shrinkage.  You need to stay within the shrinkage model context.
A: You could see ridge regression as doing the feature 'selection' in a nuanced way by reducing the size of the coefficients instead of setting them equal to zero.
You could elliminate the features with the smaller coefficients*, but it is a bit crude method.
There might be better methods like LASSO or stepwise selection which compare the coefficients and models in their totality rather than just single coefficient sizes. Especially in the case of colinearity you can have large coefficients that might not need to be an indication that the feature strongly correlates with the outcome, but to the feature cancelling the error of some other feature.**

* Be aware that the effect of a feature is not just the size of the coefficients but also the scale/size of the feature. For instance a model with features like man/woman (encode as 0/1) and income (encoded as dollars per year) might give a small coefficient for dollars per year but the effect could be large (because the feature has a larger range of values). Ridge regression or other regularisation of the coefficients makes most sense when the features are normalized or at least have similar scales. Otherwise you might better use methods that compare significance or likelihood of models with and without features.
** Colinearity might actually be a problem in ridge regression. These combinations of parameters whose difference are good features are penalized relatively stronger.
A: There are some advantages to using Ridge regression over LASSO, for ex. if you expect everything to be correlated with everything. Ridge regression has already performed variable selection for you (similar to LASSO), that is all variables with coefficients !=0 have an effect. It may happen that some variables have coefficients very close to 0 but not exactly 0, you can threshold these to 0.
Also note the meaning of "best" variables, these are variables which seem to predict well on your data, these do not necessarily have a meaningful interpretation. Also make sure you scaled all variables when building the model.
