This depends on the models (and maybe even software) you want to use. With linear regression, or generalized linear models estimated by maximum likelihood (or least squares) (in R this means using functions
glm), you need to leave out one column. Otherwise you will get a message about some columns "left out because of singularities"$^\dagger$.
But if you estimate such models with regularization, for example ridge, lasso er the elastic net, then you should not leave out any columns. The regularization takes care of the singularities, and more important, the prediction obtained may depend on which columns you leave out. That will not happen when you do not use regularization$^\ddagger$. See the answer at How to interpret coefficients of a multinomial elastic net (glmnet) regression which supports this view (with a direct quote from one of the authors of
With other models, use the same principles. If the predictions obtained depends on which columns you leave out, then do not do it. Otherwise it is fine.
So far, this answer only mentions linear (and some mildly non-linear) models. But what about very non-linear models, like trees and randomforests? The ideas about categorical encoding, like one-hot, stems mainly from linear models and extensions. There is little reason to think that ideas derived from that context should apply without modification for trees and forests! For some ideas see Random Forest Regression with sparse data in Python.
$^\dagger$ But, using factor variables, R will take care of that for you.
$^\ddagger$ Trying to answer extra question in comment: When using regularization, most often iterative methods are used (as with lasso or elasticnet) which do not need matrix inversion, so that the design matrix do not have full rank is not a problem. With ridge regularization, matrix inversion may be used, but in that case the regularization term added to the matrix before inversion makes it invertible. That is a technical reason, a more profound reason is that removing one column changes the optimization problem, it changes the meaning of the parameters, and it will actually lead to different optimal solutions. As a concrete example, say you have a categorical variable with three levels, 1,2 and 3. The corresponding parameters is $\beta_, \beta_2, \beta_3$. Leaving out column 1 leads to $\beta_1=0$, while the other two parameters change meaning to $\beta_2-\beta_1, \beta_3-\beta_1$. So those two differences will be shrunk. If you leave out another column, other contrasts in the original parameters will be shrunk. So this changes the criterion function being optimized, and there is no reason to expect equivalent solutions! If this is not clear enough, I can add a simulated example (but not today).
you end up with correlated features, so you should drop one of them as a "reference"Dummy variables or indicator variables (these are the two names used in statistics, synonymic to "one-hot encoding" in machine learning) are correlated pairwisely anyway, be they all k or k-1 variables. So, the better word is "statistically/informationally redundant" instead of "correlated". $\endgroup$
Does keeping all k values theoretically make them weaker features. No (though I'm not 100% sure what you mean by "weaker").
using something like PCANote, just in case, that PCA on a set of dummies representing one same categorical variable has little practical point because the correlations inside the set of dummies reflect merely the relationships among the category frequencies (so if all frequencies are equal all the correlations are equal to 1/(k-1)). $\endgroup$
is_malevariable as opposed to both options? Maybe that doesn't make sense in this context, and it might only be an issue when you have two different variables actually encoding the same information (e.g. height in inches and height in cm). $\endgroup$