I have a dataset of NFL games. Each game has one row for each team in the game. Each team's row contains the team's statistics in that game (such as points scored, passing yards, red zone attempts, etc.), the team's statistics over the team's previous four games, and the team's opponent's statistics over their previous four games.
So, for example, the one row might contain a team's passing yards in game
gt, that team's total passing yards over games
gt - 4 through
gt - 1, and the team's opponent's passing yards allowed over games
go - 4 through
go - 1.
What I have found is that, for some statistics, a team's previous four games' total (and possibly their opponent's previous four games' total) gives a better prediction of what they will do in a given game than a Random Forest Regressor trained on all the variables. Using the example from above:
>>> # keys beginning with 'tm_' represent the team's statistics in this game >>> # keys beginning with '_tm_' represent the team's statistics over the previous 4 games >>> # keys beginning with '_op_' represent the opponent's statistics (allowed) over the previous 4 >>> >>> from scipy.stats.stats import pearsonr >>> from sklearn.ensemble import RandomForestRegressor >>> >>> # Calculate r-squared between passing yards over the previous 4 and passing yards >>> r, p = pearsonr(df['_tm_py'], df['tm_py']) >>> r**2 0.10695998799573359 >>> >>> # Use a random forest regressor to predict passing yards using all the features in the data set >>> rgr = RandomForestRegressor(n_jobs=-1, max_features=None, n_estimators=500, oob_score=True) >>> rgr.fit(df[features], df['tm_py']) RandomForestRegressor(bootstrap=True, compute_importances=None, criterion='mse', max_depth=None, max_features=None, max_leaf_nodes=None, min_density=None, min_samples_leaf=1, min_samples_split=2, n_estimators=500, n_jobs=-1, oob_score=True, random_state=None, verbose=0) >>> rgr.oob_score_ 0.099712596456913757 >>> >>> >>> # print the top-10 most important features >>> for f, i in sorted(zip(features, rgr.feature_importances_), key=lambda x: x, reverse=True)[:10]: print f, ' \t', round(i, 4) ... _tm_py 0.0929 _tm_rzpy 0.0098 _op_py 0.008 _tm_pts 0.0075 _tm_pc 0.0071 _tm_sfpy 0.007 _tm_pfd 0.0068 _tm_td 0.0067 _op_p2y 0.0065 _tm_p2y 0.0065
As you can see, a team's passing yards over their previous four games is the most important feature for predicting their passing yards (followed by the team's red zone passing yards over their previous four games and their opponent's passing yards allowed over their opponent's previous four games). However, the
oob_score_ for the Random Forest Regressor turns out to be lower than the r-squared between
Why is this? Is it because all of the other features are somehow "polluting" the direct relationship between these two variables? Is it possible that using a linear correlation between a single feature and the target can be better than using a random forest regressor between many features and the target?