3
$\begingroup$

This is my example in Python. As you can see, RidgeCV, which performs cross-validation and picks the best alpha to use, picks the model with the smallest R squared as the best model... Why?

This is my X:

 X =  np.array([[  5.,   8.,   3.,   4.,   0.,   5.,   4.,   0.,   2.,   5.,  11.,
              3.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   3.,   5.,  13.,
              4.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   4.,   4.,   0.,   3.,   5.,  12.,
              2.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   4.,   5.,  12.,
              4.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   3.,   5.,  12.,
              5.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   2.,   4.,   0.,   3.,   5.,  13.,
              3.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   2.,   4.,   0.,   4.,   5.,  11.,
              4.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   2.,   4.,   0.,   3.,   5.,  11.,
              5.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   3.,   5.,  12.,
              5.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   3.,   5.,  12.,
              5.,  19.,   2.]])

and this is my response y

y = np.array([ 70.14963195,  70.20937046,  70.20890363,  70.14310389,
        70.18076206,  70.13179977,  70.13536797,  70.10700998,
        70.09194074,  70.09958111])

Ridge Regression

    # alpha = 0.1
    model = Ridge(alpha = 0.1)
    model.fit(X,y)
    model.score(X,y)   # gives 0.36898424479816627

    # alpha = 0.01
    model1 = Ridge(alpha = 0.01)
    model1.fit(X,y)
    model1.score(X,y)     # gives 0.3690347045143918 > 0.36898424479816627

    # alpha = 0.001
    model2 = Ridge(alpha = 0.001)
    model2.fit(X,y)
    model2.score(X,y)  #gives 0.36903522192901728 > 0.3690347045143918

    # alpha = 0.0001
    model3 = Ridge(alpha = 0.0001)
    model3.fit(X,y)
    model3.score(X,y)  # gives 0.36903522711624259 > 0.36903522192901728

Thus from here it should be clear that alpha = 0.0001 is the best option. Indeed reading the documentation it says that the score is the coefficient of determination. If the coefficient closest to 1 describes the best model. Now let's see what RidgeCV tells us

RidgeCV regression

modelCV = RidgeCV(alphas = [0.1, 0.01, 0.001,0.0001], store_cv_values = True)
modelCV.fit(X,y)
modelCV.alpha_  #giving 0.1
modelCV.score(X,y)  # giving 0.36898424479812919 which is the same score as ridge regression with alpha = 0.1

What is going wrong? Surely we can check manually, as I have done, that all the other alphas are better. So not only it is not choosing the best alpha, but it is choosing the worst!

Can someone explain to me what it's going wrong?

$\endgroup$
  • 1
    $\begingroup$ Perhaps the modelCV.score is value of the fit on the whole data but the actual CV fold scores are different (and possibly not shown)? $\endgroup$ – DataD'oh Sep 13 '17 at 11:24
5
$\begingroup$

What went wrong? You're comparing apples with oranges

  • You computed training scores, meaning you scored the model on data you trained it with. You should compute test scores, i.e. a score on data the model has not yet seen.
  • The manual check you did is just one experiment. If you repeat this several times with different $X$ and $y$, you will find model.score to give different values each time. Since you did only one experiment, i.e. used only one X and one y, your manual check is not very robust.
  • Therefore, your manual check produces a one-sample training score.

RidgeCV is doing something more robust:

  1. It takes the $X$ and $y$ you provided, splits them into 3 parts
  2. It internally does model.fit on the first 2 parts and then model.score on the 3rd part, a test score.
  3. It repeats step 2, but doing model.fit on part 2 and 3, then model.score on part 1
  4. It repeats step 2 again, but doing model.fiton part 3 and 1, then model.score on part 2
  5. Finally it averages the model.score values from the three steps before, this is what it returns as modelCV.score. That means, RidgeCV produces an averaged test score.

The above steps are repeated for all alphas you provide. Each alpha produces such an average.

The above process is called 3-fold cross validation, you can read more about it on wikipedia or the scikit-learn documentation.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you, super clear explanation! Realized only now! SO basically to make real comparisons, I should have checked cross-validation for all four alphas manually, and only then I would have found the same result as RidgeCV? $\endgroup$ – Euler_Salter Sep 13 '17 at 13:58
  • $\begingroup$ Also, Ridge CV seems to be giving R^2 (also, why do people provide R^2 and not adjR^2?) with model.score(x,y). I also could compute RMSE. If my goal is see how well the models can predict, should I use RMSE right? Or should stick with the R^2? $\endgroup$ – Euler_Salter Sep 13 '17 at 14:00
  • 1
    $\begingroup$ You should decide what is more useful to you. If you're interested in the average prediction error, then RMSE on the test set is interesting. If you're more interested in knowing how much of the variance in the data your model captures, you should go with R^2. Why sklearn uses R^2 by default is not known to me, but my guess would be because it's more common and simpler to compute. $\endgroup$ – Denwid Sep 13 '17 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.