Are there any sanity checks for high R2 value?

I am training a RandomForestRegressor Model with Scikit-Learn to model a physical process. The dataset has the following properties:

• 450’000 samples, 42 features
• Train test split of 80/20

When calculating the test score, I get a very high test R2 value of 0.985 and a test RMSE of 0.71. I did visualize the results in the plot below (y_pred vs. y_test): [UPDATE1] added figure with s=1 and alpha = 0.1  I am a bit suspicious, about the high test R2 value. Even though the plot indicates a relatively good fit, the test and prediction value don’t match “perfectly”. I don’t have much intuition about what to expect from the R2 value in this case.

Does anyone have an idea, if such a high R2 is plausible? Are there any sanity checks (next to using a test set), that I could apply to verify my results? Thank you!

• it all comes down to the nature of the data and what you know or do not know about the data generation process. In many fields $R^2 \ll 1$ is a sign of incompetence (e.g. where the issue is measurement or sampling error in experiments and only very high predictability is acceptable). In others $R^2 \sim 1$ signals tautology or fraud or out-of-control overfitting, as inherent predictability is very low. In social science it would be scary if knowing gender, race and some categories were more than weakly predictive of any kind of outcome because so much comes down to individual-level detail. May 12 '20 at 10:02
• Your plot is not too informative. How many out of these 450’000 observations do in fact have high standardized residuals? The plot may mask that these are just a very small portion since the points in your plot overlap. If you used CV to find the best parameter combination, then you need an independent, fresh data set to estimate oos-prediction accurately. May 12 '20 at 10:12
• Judging 90,000 points in a plot is difficult due to overplotting, as 00schneider has pointed out. For visual inspection you should make at least two more plots: the same as the one above but with very small dots an in a very transparent color(set alpha near zero) and you should calculate the residuals (prediction errors) and plot a histogram or density plots of how those are distributed. May 12 '20 at 10:29
• @00schneider, so far, I just did a train/test split of the data, standardized the values (only using the training set) and fit the model on the training set without hyperparameter tuning. Finally, did evaluate the model on the test set.
– lux7
May 12 '20 at 11:31
• When fit is very good a plot like this is quite reassuring, and good propaganda, but it is best to switch to a plot of residual versus predicted or fitted, to see better the structure the model is not capturing. May 12 '20 at 12:29