What is the best practise when it comes to choosing how many models to evaluate when building a predictive model? It seems there are countless possibilities so I'm not sure how one chooses where to draw the line.

Assume we had a set of data that contained thousands of samples with the following information: gender, age, height, weight, country.

Now, suppose we wanted to build a model for predicting people's heights based on gender, age, weight, and country.

I have begun considering a multiple linear regression approach to this problem. But would it be a bad practise to only consider multiple linear regression? Is the best practise to also consider support vector regression, polynomial regression, ridge regression, etc.? We could sub-categorize even further as we could try many different orders in polynomial regression, or many different regularization parameters in ridge regression.

How would an experienced statistician/data scientist approach the model selection component of this problem?

  • $\begingroup$ Figure out what out of sample criterion interests you and cross validate. $\endgroup$ – gung - Reinstate Monica Sep 11 '20 at 19:20

First off, I think it is useful to start looking at generalizations of models rather than their exact implementations. A polynomial regression is a regression just with a polynomial expansion (feature engineering) and multiple linear regression is a simple linear regression just with a different input matrix size. It's really all the same "model" and lives nicely under a GAM setup. If we go into Bayesian land then we can include ridge/lasso via setting different priors. Of course there are different assumptions caked in to all of that but I think you should think bigger picture like a linear model vs a tree model vs a neural net. Think of it like model families.

Now, for prediction I normally start off with a linear model as a baseline. Then you can do a simple polynomial expansion and throw on regularization to get a good idea of what a linear model may be capable of. Then I go into boosted trees. Normally boosted trees give me by far the best predictive performance. There is a reason why it has dominated the data science competition scene for 5 years.

Now the question is WHY would a boosted tree give you better performance. The answer usually lies with how it handles data interaction and natural dynamics across a variable.

For your example, would someone have a different rate of growth from ages 1-20 than from 20-50? Yes, and we would have to add that knowledge into a linear model or use the GAM setup. Now, do you think age AND gender have an interaction together? Once again, we have to inject that knowledge with a linear model whereas a tree naturally handles that and handles the information better than a GAM setup (90% of the time).

So where does a linear model do better? Well, let's say we want to predict the price of a stock only given past prices. How would a tree handle a structure such as a trend in this time series? It really can't unless we change it to like an auto regressive tree, SO I would invest more time in a linear setup or RNN/LSTM. Alternatively, if I only have 50 observations, we typically can't 'learn' too much anyway so a linear regression may just be the best setup.

Your time is the most valuable resource, so you should be going towards model paradigms which are proven to do better in that field. Setup some basic models for baseline and iterate the better performing model 'families'.

  • $\begingroup$ Excellent answer, thanks mate! I will look into boosted trees over the next few days. If we take the boosted tree approach is it possible to calculate things like confidence bounds on our predictions? So we can quantify the uncertainty. Or does the boosted tree approach provide us with very good pointwise predictions without a quantification of the uncertainty? $\endgroup$ – ManUtdBloke Sep 8 '20 at 9:11
  • 1
    $\begingroup$ Yeah, libraries like Xgboost and Lightgbm give you the option to use different cost functions, one of which is 'quantile'. The parameter is what quantile you want so for example .9 would be a 90% probability that the point lies below which can be your upper limit and .1 would be 10% or a lower limit. However, you usually get your best point wise accuracy by using the standard regression cost function. There actually was a lot of great work done on a recent Kaggle event for the M5 competition I highly recommend you snoop around! $\endgroup$ – Tylerr Sep 8 '20 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.