# Machine learning models for regression on small data sets

What are the "best" models to be used for simple regression of 1 numerical variable using only a small data set of e.g. 250 samples and up to 10 features?

I understand that the data set is super small (even smaller if one applies e.g. a 60%/40% train-test split) and that this carries a high risk of over-fitting especially when using complex models like neural networks.

What would be a reasonable model to use in such a case and what would be the best way to avoid over-fitting? Note that I do not know if relationships are linear or if all features are necessarily helpful.

• "Simple regression" is a linear regression with single feature, so I guess you meant linear regression. However linear regression already is a model, so what exactly do you mean by "models"? – Tim May 27 '19 at 17:42
• Thanks Tim. I meant a model that predicts a single (continuous, real-valued) variable based on the data available. Linear regression would indeed be one possible model, among other models. The ~250 samples are daily observations. – robot_2077198 May 28 '19 at 15:43

Small datasets and few features are a domain where traditional statistical models tend to do very well, because they offer the ability to actually interpret the importance of your features.

I'm assuming by "simple regression" you mean predicting a real-valued, continuous variable y from your input variables. You mention that you suspect you may have non-linear relationships, and you don't know much about the importance of the features.

My instincts in this case would be to use a generalized additive model (GAM), like the mgcv package for R. mgcv has very nice default methods for choosing some of the more arcane parameters in a GAM, like how many knots and where to put them.

Maybe you have three predictors, x1, x2, and x3, where x1 and x2 are continuous and x3 is a categorical variable. In this case you could do (in R):

library(mgcv)
x3 <- as.factor(x3)
my.model <- gam(y ~ s(x1) + s(x2) + x3, method = "REML")
summary(my.model)


That last part about using REML is personal preference. It sets how "wiggly" the nonlinear curves are allowed to be. The default method uses, if I recall, generalized cross-validation, which works fine though in my experience tends to give "wigglier" curves.

The problem with daily data and only 250 days is that you could face seasonality issues that you can't really evaluate statistically, only by business knowledge.

But regardless of seasonality, 250 samples and 10 features are quiet enough in my opinion to build a predictive modeling.

The best way to do it is use boosted regression (see xgboost, does a great job right now, very popular and easy to understand) with a good validation process like this one I use a lot right now on small datasets: http://dataneel.github.io/nx2_cross_validation/

I'm not a fan of doing regularization on 10 features only, really not necessary, you can have the luxury to study the impact of each variable on your target with basics analysis of correlation, plot each x with your y to see the shape of the relationship, ....

If you are considering linear models, and are concerned with overfitting, you can consider using linear regression with regularization ie. ridge regression or lasso or a combination("elastic net").

If you want to try out non-linear as well as interaction terms, you can try SVM regression with a polynomial kernel or an RBF kernel.

This will still require you to divide the data into train-tune-test pieces, but, you can use k-fold cross-validation for "train-tune" part (to trade-off lack of data for additional computation). You can keep 25% for test.

It difficult to avoid a test sample if you are concerned about overfitting - this is because unless you test the fit model on an unseen sample, you can't get an unbiased estimate of model's performance.

With such a small data set, I would consider a few options:

1. Neural network with some transfer learning if you can find a larger well-labeled data set to complement what you have
2. A semi-supervised approach if you can find a large source of unlabeled data to complement your data
3. Bayes net if you are able to reasonably hand craft some priors