Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have modelled a lasso regression for a set of features. The model looks suspiciously very good. The $R^2$ is about $0.98$ and the $RMSE$ is $0.03$ for the $y$ (the predicted value) ranges from 1 to 3000.
Such good result makes me feel that there is something wrong, given that I did testing on a separate set that was never used in the method (except in the testing)
how can I make sure that the model is really good and there is nothing done wrongly.
If you have truly used a separate training set, then things should be fine. Some things can simply be modeled and predicted rather well. (See astronomy. We are really good at predicting where Jupiter will be in a few months' time. Which is good, because otherwise, probes would miss it.)
Of course, there are a few caveats. For instance, perhaps you ran hundreds of models, each of them modeled on the training data and evaluated on separate test data - and now you wonder why the top performing model is so very good. This would simply be a case of "overfitting to the test set", and of course, you shouldn't expect this good performance with truly new data.
Or perhaps you used a predictor that is in fact only available when you have your new test data. For instance, a colleague of mine recently got extremely good forecasts when forecasting the number of units sold in a retail store. He was suspicious, and it turned out that he had inadvertently included dollar sales as a predictor - which is of course highly correlated with unit sales, but not available before the unit sales are.
Similarly, I once improved my sales forecasts by an incredible amount. Then I realized that one of my predictors, All Commodity Volume (ACV), was essentially an aggregate of the number I was forecasting, and would of course not be available ahead of time for actual forecasting.
People sometimes use weather information in improving sales forecasts. Which is nice and good - but they should really use weather forecasts, not actual weather, because the actual weather two days out is not known yet when we forecast sales two days out. An error like this can make your predictions look far better than they will truly be in a production environment.
(Incidentally, in German this is known as sich in die eigene Tasche lügen, "lying into one's own pocket".)
Thus, I'd look at whether your predictors are really "honest", or whether you did any inadvertent data snooping.
