First, the question of "what drives the outcome the most" is very tricky and can be poorly posed. Consult this page and its links before you proceed.
Second, binary regression has inherent omitted-variable bias. If you omit any outcome-associated predictor, you risk bias in the coefficient estimates for the included predictors. It's usually a good strategy to include as many predictors as reasonable without overfitting.
Third, your data might be adequate to fit without penalization. As discussed in comments, that depends on the number of coefficients you need to estimate and the number of cases in the minority class. For a multi-level categorical predictor you need to estimate coefficients for one less than the number of levels. A useful rule of thumb is that about 15 minority-class cases per coefficient is adequate to avoid serious overfitting. See Chapter 4 of Frank Harrell's course notes or book.
Fourth, if you are in danger of overfitting, first apply data-reduction techniques combined with your understanding of the subject matter to reduce the dimension of your predictor space without looking at the outcomes. The Harrell chapters cited above contain many suggestions for things like combining related predictors, removing predictors with narrow distributions, combining related levels of categorical predictors, principal-components analysis (or related techniques), etc.
Fifth, you need to avoid focusing on the individual predictor coefficients reported for multi-level categorical predictors. Each typically represents the significance of a difference from whatever level you chose as reference, so "significance" of individual coefficients depends on that choice. You want an overall estimate that combines all levels, for example a Wald test on all coefficients together. See Section 5.4 of either of the Harrell references for how to do this, in expositions of why such attempts to evaluate predictor importance can be difficult or misleading .
Related to that problem, with standard lasso your final model might keep some levels of a categorical predictor while omitting others. You need the group lasso to keep sets of related coefficients together.
Sixth, if you still need to use a penalized approach like ridge or lasso, then you won't be able to use the simplest out-of-the-box measures with your multi-level categorical predictors. The default is typically to normalize all predictors to zero mean and unit standard deviation so that penalization of coefficients for continuous predictors doesn't depend on whether distances, say, are measured in millimeters or miles. That approach has problems even with binary predictors; with multi-category predictors, the choice of reference level can change the penalization! See this page and its links.
