1
$\begingroup$

I've been trying to learn some of the basics of machine learning. As a test project, I'm trying to predict NFL wide receivers fantasy football performance by season using data on: their past performance, their team's past performance, and measures of their teammates' ability. My goal is to find a model that predicts player performance better than players' average draft position (ADP) in real fantasy drafts.

I have data from 2000-2019. My full dataset has 1843 observations of almost 200 variables. Note that I use ADP as a variable.

My approach has been to train a random forest model, with cross-validation, on the 2000-2018 data and then measure model performance when predicting on the 2019 data.

The model I used was:

(model0018 <- train(
  fantasy_points ~ .,
  data0018,
  method = "ranger",
  metric = "MAE",
  tuneLength = 5,
  trControl = trainControl(
    method = "cv",
    number = 5,
    verboseIter = TRUE
  )
))

The model does not yield good results. My model's predictions only beat ADP 40% of the time for the 2019 data, and that's using ADP as a predictor variable.

When I include 2019 observations in the training set, however, the model predicts these 2019 observations very well. It beats ADP's predictions 84% of the time.

After observing this poor out-of-sample, but strong in-sample performance, I reasoned that my model must be overfitting the data. So I reduced the number of predictors to just 4 predictors I thought would be most relevant.

The model still works very well in-sample, beating the ADP predictions 70% of the time. However, its performance out-of sample was still much worse - beating ADP predictions only 53% of the time.

How is it possible that such a simple model, with 460 observations to every predictive variable, is still overfitting? What can I do to try to fix this problem?

$\endgroup$
0

1 Answer 1

0
$\begingroup$

Overfitting is a symptom of a complex model memorizing data.

If you are you tuning hyperparameters (and thus giving your model the option to grow in complexity) on data that occurs during the same period of time on which you are training this may be occurring. IE you state you are training and validating on 2000-2018. Hold out several years for validation.

If this still doesn't yield acceptable results consider a model designed for sequential data like an RNN.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.