# What is the best regression method for small data sets (e.g., ~1000-5000 data points)?

I find it quite hard to understand what is the state of the art method for a regression task involving a small data set (e.g., ~1000-5000 data points). In some books, we can find linear regression, tree based algorithms, support vector machines, etc. I am wandering if there is now a common agreement on which method has the most benefit.

## 2 Answers

The best regression method depends on the data generating process.

Regardless of sample size, if your response is linearly related with the covariates, then linear regression should always be best.

If the relationship is complex but smooth, and you think uncertainty quantification is important, GP regression might be best

If the relationship is complex and uncertainty quantification is not important, a neural net might be best

I really hate to say it, but the answer is "it depends"

The size of the dataset has relatively little relevance to the question of "what type of regression model should I use."

In essence this is like asking: "If I'm only building a "small" house - should I use a philips head screwdriver or a flat head screwdriver?" The type of screwdriver you should use depends on the kind of screws you are using, not the size of the house you are building.

Different kinds of models are used to estimate/predict different types of dependent variables (e.g. a logit model for a binary variable, a linear model for a continuous one), different data structures (e.g. a multilevel models for data which are "nested" at multiple levels), and different types of research questions and different assumptions (e.g. a Targeted Maximum Likelihood Estimation" method for causal inference, given certain assumptions about the factors associated with treatments). Certainly different kinds of models often have different assumptions the sample size under which they produce reliable results, but that's secondary to the question of whether you are using the "right tool for the job."

Also, while "small" is a subjective term, I would not personally use that term to describe a dataset with several thousand observations...although this is perhaps due to my bias as someone who uses "old school" statistics for hypothesis testing and estimation as opposed to machine learning.