Best practise for model selection when building predictive models?

Question

What is the best practise when it comes to choosing how many models to evaluate when building a predictive model? It seems there are countless possibilities so I'm not sure how one chooses where to draw the line.

Assume we had a set of data that contained thousands of samples with the following information: gender, age, height, weight, country.

Now, suppose we wanted to build a model for predicting people's heights based on gender, age, weight, and country.

I have begun considering a multiple linear regression approach to this problem. But would it be a bad practise to only consider multiple linear regression? Is the best practise to also consider support vector regression, polynomial regression, ridge regression, etc.? We could sub-categorize even further as we could try many different orders in polynomial regression, or many different regularization parameters in ridge regression.

How would an experienced statistician/data scientist approach the model selection component of this problem?

Answer 1

First off, I think it is useful to start looking at generalizations of models rather than their exact implementations. A polynomial regression is a regression just with a polynomial expansion (feature engineering) and multiple linear regression is a simple linear regression just with a different input matrix size. It's really all the same "model" and lives nicely under a GAM setup. If we go into Bayesian land then we can include ridge/lasso via setting different priors. Of course there are different assumptions caked in to all of that but I think you should think bigger picture like a linear model vs a tree model vs a neural net. Think of it like model families.

Now, for prediction I normally start off with a linear model as a baseline. Then you can do a simple polynomial expansion and throw on regularization to get a good idea of what a linear model may be capable of. Then I go into boosted trees. Normally boosted trees give me by far the best predictive performance. There is a reason why it has dominated the data science competition scene for 5 years.

Now the question is WHY would a boosted tree give you better performance. The answer usually lies with how it handles data interaction and natural dynamics across a variable.

For your example, would someone have a different rate of growth from ages 1-20 than from 20-50? Yes, and we would have to add that knowledge into a linear model or use the GAM setup. Now, do you think age AND gender have an interaction together? Once again, we have to inject that knowledge with a linear model whereas a tree naturally handles that and handles the information better than a GAM setup (90% of the time).

So where does a linear model do better? Well, let's say we want to predict the price of a stock only given past prices. How would a tree handle a structure such as a trend in this time series? It really can't unless we change it to like an auto regressive tree, SO I would invest more time in a linear setup or RNN/LSTM. Alternatively, if I only have 50 observations, we typically can't 'learn' too much anyway so a linear regression may just be the best setup.

Your time is the most valuable resource, so you should be going towards model paradigms which are proven to do better in that field. Setup some basic models for baseline and iterate the better performing model 'families'.