I am aware of the fact that categorical variables with k levels should be encoded with k-1 variables in dummy encoding (similarly for multi-valued categorical variables). I was wondering how much of a problem does a one-hot encoding (i.e. using k variables instead) over dummy encoding for different regression methods, mainly linear regression, penalized linear regression (Lasso, Ridge, ElasticNet), tree-based (random forests, gradient boosting machines).
I know that in linear regression, multi-collinearity problems occur (even though in practice I have fitted linear regression using OHE without any issues).
However, does dummy encoding need to be used in all of them and how wrong would the results be if one uses one-hot encoding?
My focus is on prediction in regression models with multiple (high-cardinality) categorical variables, so I am not interested in confidence intervals.