I am approaching machine learning and I want to ask you some questions, hoping that they do not sound too trivial.
I have a pretty small dataset with $n \simeq 500$ observations and about $p \simeq 20$ non-ordered categorical features. My aim is to perform binary classification. (I mainly will use the R package caret).

In my case, how do I know what are the possible binary classification models to choose and try?
More in detail, should I have to exclude any model a priori, due to the fact, for example, that my features are categorical or due to dimensions?
Once I have a list of possible models to try, how do I know how to encode categorical features?

I appreciate any books, links or articles suggestions to study in deep that; thanks!

  • $\begingroup$ Seems sensible to start with a binomial GLM, using the base glm function. $\endgroup$
    – user19904
    Commented Dec 1, 2022 at 10:05

1 Answer 1


It's always useful to have a simple baseline model such as a logistic regression (possibly in its penalized LASSO/elastic net form), in which categories you want to deal with are either dummy-coded fixed effects, or perhaps instead random effects (also makes it easier to deal with new unseen categories later on). I'd start there. Additionally, make sure to have a good evaluation approach to see what models really improve vs. what is overfitting (e.g. good cross-validation, plans for prospective evaluation on new data).

Thereafter, there are a lot of things you can try. Even algorithms that do not naturally deal with categorical data can of course do so e.g. via target encoding (which can be done even for algorithms that have ways of dealing with categories, some algorithms even effectively do a form of target encoding in the background such as catboost, while you could look at random effects for categories in random effects logistic regression as a form of it, too). Other algorithms like neural networks have explicit ways of dealing with categories such as embedding layers (you can also try to use the outputs of embedding layers as inputs for other algorithms in order to represent categories) - with many categories (think problems like predicting which of the thousands of films the hundred thousands of users of a streaming platform would like, or representing products users might buy on an online platform) embedding layers tend to be one of the best performing approaches and neural networks become extremely competitive with other approaches (this may be less the case in your low dimensional case). I don't think I would a-priori exclude any particular algorithm, some might just need more regularization/shrinkage (e.g. very few layers + very few neurons per layer + weight decay for neural networks) in order to not overfit on such a small dataset.

In terms of reading material, I would look at what people tend to do in Kaggle competitions. Some good introductory reading is e.g. Kaggle competition GM Thakur's book, the Kaggle book, the book on the fast.ai course and there's an excellent How to Win a Data Science Competition: Learn from Top Kagglers course on coursera.org (but that's currently inaccessible, if you have not already enrolled, due to the association of the course with Moscow university).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.