You don't provide a minimal reproducible example, so there is room for one more guess about the issue you report.
In a comment you say that you are working with 100 data points. So when you do 3-fold cross validation you end up training the model on 66, 67 and 67 points and evaluating it on another 34, 33 and 33 data points.
Obviously that's not much data to train a model on. The polynomial features compound the issue because polynomials are bad at extrapolating outside of the observed range.
So my stab at what's happening is that: (a) your data is small, so the in-fold range and the out-fold range of the predictor(s) end up being "different enough"; (b) you use polynomials which are bad at extrapolating outside of the range of the predictor(s) observed during training; and (c) you compound the problem with variability/instability by using a different 3-fold split for each polynomial degree.