What are the "best" models to be used for simple regression of 1 numerical variable using only a small data set of e.g. 250 samples and up to 10 features?

I understand that the data set is super small (even smaller if one applies e.g. a 60%/40% train-test split) and that this carries a high risk of over-fitting especially when using complex models like neural networks.

What would be a reasonable model to use in such a case and what would be the best way to avoid over-fitting? Note that I do not know if relationships are linear or if all features are necessarily helpful.

  • 2
    $\begingroup$ "Simple regression" is a linear regression with single feature, so I guess you meant linear regression. However linear regression already is a model, so what exactly do you mean by "models"? $\endgroup$ – Tim May 27 '19 at 17:42
  • $\begingroup$ Thanks Tim. I meant a model that predicts a single (continuous, real-valued) variable based on the data available. Linear regression would indeed be one possible model, among other models. The ~250 samples are daily observations. $\endgroup$ – robot_2077198 May 28 '19 at 15:43

Small datasets and few features are a domain where traditional statistical models tend to do very well, because they offer the ability to actually interpret the importance of your features.

I'm assuming by "simple regression" you mean predicting a real-valued, continuous variable y from your input variables. You mention that you suspect you may have non-linear relationships, and you don't know much about the importance of the features.

My instincts in this case would be to use a generalized additive model (GAM), like the mgcv package for R. mgcv has very nice default methods for choosing some of the more arcane parameters in a GAM, like how many knots and where to put them.

Maybe you have three predictors, x1, x2, and x3, where x1 and x2 are continuous and x3 is a categorical variable. In this case you could do (in R):

x3 <- as.factor(x3)
my.model <- gam(y ~ s(x1) + s(x2) + x3, method = "REML")
plot(my.model, shade=TRUE, pages=1)

That last part about using REML is personal preference. It sets how "wiggly" the nonlinear curves are allowed to be. The default method uses, if I recall, generalized cross-validation, which works fine though in my experience tends to give "wigglier" curves.


The problem with daily data and only 250 days is that you could face seasonality issues that you can't really evaluate statistically, only by business knowledge.

But regardless of seasonality, 250 samples and 10 features are quiet enough in my opinion to build a predictive modeling.

The best way to do it is use boosted regression (see xgboost, does a great job right now, very popular and easy to understand) with a good validation process like this one I use a lot right now on small datasets: http://dataneel.github.io/nx2_cross_validation/

I'm not a fan of doing regularization on 10 features only, really not necessary, you can have the luxury to study the impact of each variable on your target with basics analysis of correlation, plot each x with your y to see the shape of the relationship, ....


If you are considering linear models, and are concerned with overfitting, you can consider using linear regression with regularization ie. ridge regression or lasso or a combination("elastic net").

If you want to try out non-linear as well as interaction terms, you can try SVM regression with a polynomial kernel or an RBF kernel.

This will still require you to divide the data into train-tune-test pieces, but, you can use k-fold cross-validation for "train-tune" part (to trade-off lack of data for additional computation). You can keep 25% for test.

It difficult to avoid a test sample if you are concerned about overfitting - this is because unless you test the fit model on an unseen sample, you can't get an unbiased estimate of model's performance.


With such a small data set, I would consider a few options:

  1. Neural network with some transfer learning if you can find a larger well-labeled data set to complement what you have
  2. A semi-supervised approach if you can find a large source of unlabeled data to complement your data
  3. Bayes net if you are able to reasonably hand craft some priors
  4. Adding regularization to your model

It's tough to say exactly what is best without understanding the dataset and the operating goals of your model. Typically, you're better off spending your effort improving your data sources than your model with this little data though if you hope to generalize well in the real world

  • 1
    $\begingroup$ Neural network with 250 samples, are you sure about this proposition..? $\endgroup$ – Tim May 27 '19 at 20:05
  • $\begingroup$ Thanks Brent. Unfortunately the data is really contained to the set of ~250 samples with no possibility to obtain more labeled data or use large sources of unlabeled data (for semi-supervised learning). Regularisation will be a necessary requirement to avoid overfitting any model on such a small data set. As Tim said, I would not be inclined to train neural networks (even shallow ones) on such small data. $\endgroup$ – robot_2077198 May 28 '19 at 15:47
  • $\begingroup$ I agree with you and Tim that neural network are a poor choice with no access to tangential additional data. I like Bayesian models with little data if you have domain expertise, incorporating priors can help with lack of data if used wisely. If this isn't possible, John is spot on that traditional methods are the way to go. Best of luck! $\endgroup$ – Brent May 29 '19 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.