SVR optimal hyperparameters are Epsilon = 0, Cost = inf?

Question

I'm running an rbf-kernel SVR with GridSearchCV. I'm optimizing epsilon, cost and gamma. In my hyperparameter gridsearch, the optimal parameters appear "unbounded". Specifically, any epsilon under 1 seems to work equally well - even an epsilon of 0. And within this range, the cost can be anywhere from .5 to 10000000 without hardly changing the result. See heatmap of TEST r2 values at a representative epsilon (.5) . I'm concerned that this unboundedness might indicate a deeper problem. It feels like I'm overfitting (low epsilon high cost), although it's optimized on test r2 so I'm not sure how that's possible.

Thank you for any insight!

Answer 1

I'll answer my question with what I learned, incomplete as it may be. First, I was running my SVR with thousands of input parameters. Reducing my features to ~100 by doing feature selection (specifically, Mutual Information) helped a lot with correcting this "unbounded" hyperparameter problem.

Second, plateaus in the hyperparameters are apparently not generally frowned upon - it is considered reasonable to find the "start" of the plateau and use that set of hyperparameters (so in the above heatmap, C=1 gamma=.1).