3
$\begingroup$

This question has been asked before, but I have a specific case I'm curious about. Essentially, I have a very low MSE, but the R^2 turns out to be negative. Even the scatterplot shows that a horizontal line isn't a good fit, so I'm not sure what to make of these results. I used sklearn's Linear Regression object, and calculated "Variance Score" (R^2) with the .score() function with .score(Xtrain, ytrain) for the training set (leftmost plot) and .score(Xtest, ytest) for the testing set (middle plot). These are time series predictions btw Thanks![enter image description here]1 [enter image description here]2

$\endgroup$
1
  • $\begingroup$ Welcome to Cross Validated! Please take a moment to view our tour. It looks like you end up with a combinations of two distributions (or a bi-modal distribution) which is especially prevalent in the second model. $\endgroup$
    – Tavrock
    Mar 13, 2017 at 19:35

1 Answer 1

1
$\begingroup$

The $R^2$ value indicates that a null model (horizontal line) is a better fit for the test data than the model you've created in the training data. If you plot the $x=y$ line in your test set graphs, you can see that this line lies almost entirely above the predicted y values. For the null model, roughly half of the predicted values will lie above this line and half will lie below it, even though the predictions are all the same.

There are two aspects here that perhaps need looking at:

  1. The training data and test data appear to be different (the range of y values are certainly very different). Why is this? Should they be expected to be described by the same model? Have they come from the same data generation process?

  2. Being able to fit the null model for the test data requires knowing the average true y value across the test data. A more appropriate null model in this case would be the average training data y value. Because of the different ranges of the training and test data, your model would likely do much better than this (but remain a bad model for the test data)

The small $MSE$ doesn't indicate much here. The null model would also have a small $MSE$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.