2
$\begingroup$

I have modelled a lasso regression for a set of features. The model looks suspiciously very good. The $R^2$ is about $0.98$ and the $RMSE$ is $0.03$ for the $y$ (the predicted value) ranges from 1 to 3000.

Such good result makes me feel that there is something wrong, given that I did testing on a separate set that was never used in the method (except in the testing)

how can I make sure that the model is really good and there is nothing done wrongly.

$\endgroup$
3
$\begingroup$

If you have truly used a separate training set, then things should be fine. Some things can simply be modeled and predicted rather well. (See astronomy. We are really good at predicting where Jupiter will be in a few months' time. Which is good, because otherwise, probes would miss it.)

Of course, there are a few caveats. For instance, perhaps you ran hundreds of models, each of them modeled on the training data and evaluated on separate test data - and now you wonder why the top performing model is so very good. This would simply be a case of "overfitting to the test set", and of course, you shouldn't expect this good performance with truly new data.

Or perhaps you used a predictor that is in fact only available when you have your new test data. For instance, a colleague of mine recently got extremely good forecasts when forecasting the number of units sold in a retail store. He was suspicious, and it turned out that he had inadvertently included dollar sales as a predictor - which is of course highly correlated with unit sales, but not available before the unit sales are.

Similarly, I once improved my sales forecasts by an incredible amount. Then I realized that one of my predictors, All Commodity Volume (ACV), was essentially an aggregate of the number I was forecasting, and would of course not be available ahead of time for actual forecasting.

People sometimes use weather information in improving sales forecasts. Which is nice and good - but they should really use weather forecasts, not actual weather, because the actual weather two days out is not known yet when we forecast sales two days out. An error like this can make your predictions look far better than they will truly be in a production environment.

(Incidentally, in German this is known as sich in die eigene Tasche lügen, "lying into one's own pocket".)

Thus, I'd look at whether your predictors are really "honest", or whether you did any inadvertent data snooping.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you so much for the explanation. Do splitting the data set randomly into two sets: Training set and Testing set considered "truly used a separate training set"? The Testing set was never used in neither the training nor the validating the model. $\endgroup$ – M.M Jun 10 '17 at 14:44
  • 1
    $\begingroup$ The ideal would be looking at your test set once and once only. If you looked at your test set only once and got an RMSE of 0.03, then things seem to be fine (up to the inadvertent snooping I discussed). If you fitted model 1, evaluated it on the test set, fitted model 2, evaluated it on the test set, up to model 1,000, then chose the model that performed best on the test set, then you looked at the test set not once, but 1,000 times, and you will probably have "overfitted on the test set". $\endgroup$ – Stephan Kolassa Jun 10 '17 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.