Why my R^2 for my cross validation decreases as the polynomial degree increases?

Question

Trying to do CV for my polynomial regressor. However, for some polynomial degrees:as the polynomial degree increases, R^2 decreases (e.g., R^2 for degree 2 is 0.6 while for degree 3 is 0.28), why is that?

lin_regressor = LinearRegression()

# pass the order of your polynomial here  
poly = PolynomialFeatures(1)

# convert to be used further to linear regression
X_transform = poly.fit_transform(x_train)

# fit this to Linear Regressor
linear_regg=lin_regressor.fit(X_transform,y_train)

linear_regg.coef_

from sklearn.model_selection import cross_val_score
    from sklearn.model_selection import KFold
    from sklearn.metrics import r2_score
    
crossvalidation_poly = KFold(n_splits=3, shuffle=True) 

#for train_index, test_index in crossvalidation_poly.split(X_normalized):


for i in range(1,11):
    poly_cross_validation = PolynomialFeatures(degree=i)
    X_current = poly.fit_transform(X_normalized)
    model = lin_regressor.fit(X_current, y_for_normalized)
    scores = cross_val_score(model, X_current,y_for_normalized, scoring='r2', cv=crossvalidation_poly,
 n_jobs=1)
    

    print("\n\nDegree-"+str(i) +" polynomial: R^2 for every fold: " + str(np.abs(scores)))
          
          #+" training: " + str(np.abs(train_index))+" \ntesting: " + str(np.abs(test_index)))

    print('\033[1m'+"Degree-"+str(i)+ '\033[1m'+ " polynomial: Average R^2 for all the folds: " + str(np.mean(np.abs(scores))) + '\033[0m'+ ", STD: " + str(np.std(scores)))


Degree-1 polynomial: R^2 for every fold: [0.41300831 0.45801624 0.17011995]
Degree-1 polynomial: Average R^2 for all the folds: 0.34704816498535956, STD: 0.2860884371794798


Degree-2 polynomial: R^2 for every fold: [0.75123033 0.85035531 0.40642591]
Degree-2 polynomial: Average R^2 for all the folds: 0.6693371814650284, STD: 0.19025980734977752


Degree-3 polynomial: R^2 for every fold: [0.30689692 0.1496736  0.38827092]
Degree-3 polynomial: Average R^2 for all the folds: 0.28161381160006743, STD: 0.23675178460286633


Degree-4 polynomial: R^2 for every fold: [0.7209975  0.40749117 0.84886534]
Degree-4 polynomial: Average R^2 for all the folds: 0.6591180032208857, STD: 0.18542670407038087

Answer 1

Tough to say for sure, but I anticipate the model may be overparameterized and hence have poor out of sample generalizability.

This is easy to see with a small example. Here, I've generated data from a degree 3 polynomial and cross validated over the degree for PolynomialFeatures. Here are the results of a 10 fold cross validation on rsquared

As you can see, as the degree increases sufficiently, the r squared declines much like in your example. This has to do with the bias variance trade off. As we add more parameters to the model (higher degree) the model becomes more variable as it loses bias (less biased in so far as it can now represent a broader class of functions). You're seeing the effects of this variability.

Answer 2

You don't provide a minimal reproducible example, so there is room for one more guess about the issue you report.

In a comment you say that you are working with 100 data points. So when you do 3-fold cross validation you end up training the model on 66, 67 and 67 points and evaluating it on another 34, 33 and 33 data points.

Obviously that's not much data to train a model on. The polynomial features make it even more challenging because polynomials are bad at extrapolating outside of the observed range.

So my stab at what's happening is that: (a) your data is small, so the in-fold range and the out-fold range of the predictor(s) end up being "different enough"; (b) you use polynomials which are bad at extrapolating outside of the range of the predictor(s) observed during training; and (c) you compound the problem with the intrinsic variability/instability of high-degree polynomials by using a different 3-fold split for each polynomial degree.

PS: There is another issue with how you do cross validation: You normalize the entire training data first, using all 100 points. However, the normalization step (I assume this is mean 0, variance 1 scaling?) is part of the modeling pipeline, so it should be cross validated as well.