I've been trying to learn some of the basics of machine learning. As a test project, I'm trying to predict NFL wide receivers fantasy football performance by season using data on: their past performance, their team's past performance, and measures of their teammates' ability. My goal is to find a model that predicts player performance better than players' average draft position (ADP) in real fantasy drafts.
I have data from 2000-2019. My full dataset has 1843 observations of almost 200 variables. Note that I use ADP as a variable.
My approach has been to train a random forest model, with cross-validation, on the 2000-2018 data and then measure model performance when predicting on the 2019 data.
The model I used was:
(model0018 <- train(
fantasy_points ~ .,
data0018,
method = "ranger",
metric = "MAE",
tuneLength = 5,
trControl = trainControl(
method = "cv",
number = 5,
verboseIter = TRUE
)
))
The model does not yield good results. My model's predictions only beat ADP 40% of the time for the 2019 data, and that's using ADP as a predictor variable.
When I include 2019 observations in the training set, however, the model predicts these 2019 observations very well. It beats ADP's predictions 84% of the time.
After observing this poor out-of-sample, but strong in-sample performance, I reasoned that my model must be overfitting the data. So I reduced the number of predictors to just 4 predictors I thought would be most relevant.
The model still works very well in-sample, beating the ADP predictions 70% of the time. However, its performance out-of sample was still much worse - beating ADP predictions only 53% of the time.
How is it possible that such a simple model, with 460 observations to every predictive variable, is still overfitting? What can I do to try to fix this problem?
