I've been trying to learn some of the basics of machine learning. As a test project, I'm trying to predict NFL wide receivers fantasy football performance by season using data on: their past performance, their team's past performance, and measures of their teammates' ability. My goal is to find a model that predicts player performance better than players' average draft position (ADP) in real fantasy drafts.
I have data from 2000-2019. My full dataset has 1843 observations of almost 200 variables. Note that I use ADP as a variable.
My approach has been to train a random forest model, with cross-validation, on the 2000-2018 data and then measure model performance when predicting on the 2019 data.
The model I used was:
(model0018 <- train( fantasy_points ~ ., data0018, method = "ranger", metric = "MAE", tuneLength = 5, trControl = trainControl( method = "cv", number = 5, verboseIter = TRUE ) ))
The model does not yield good results. My model's predictions only beat ADP 40% of the time for the 2019 data, and that's using ADP as a predictor variable.
When I include 2019 observations in the training set, however, the model predicts these 2019 observations very well. It beats ADP's predictions 84% of the time.
After observing this poor out-of-sample, but strong in-sample performance, I reasoned that my model must be overfitting the data. So I reduced the number of predictors to just 4 predictors I thought would be most relevant.
The model still works very well in-sample, beating the ADP predictions 70% of the time. However, its performance out-of sample was still much worse - beating ADP predictions only 53% of the time.
How is it possible that such a simple model, with 460 observations to every predictive variable, is still overfitting? What can I do to try to fix this problem?