# How to interpret the output of cross-validation for SVR

I wrote this code to run a SVR with cross validation:

def run_SVR(xTrain,yTrain,xTest,yTest,output_file,data_name):
'''
run SVR algorithm
'''

cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)

# define the pipeline to evaluate
model = SVR()
fs = SelectKBest(score_func=mutual_info_regression)
pipeline = Pipeline(steps=[('sel',fs), ('svr', model)])

# define the grid
grid = dict()
grid['sel__k'] = [i for i in range(1, xTrain.shape[1]+1)]
search = GridSearchCV(
pipeline,
param_grid={
'svr__C': [0.01, 0.1, 1, 10, 100, 1000], ##Regularization
'svr__epsilon': [0.0001, 0.001, 0.01, 0.1, 1, 10],
'svr__gamma': [0.0001,  0.001, 0.01, 0.1, 1, 10],
},
scoring='neg_mean_squared_error',
return_train_score=True,
verbose=1,
cv=5,
n_jobs=-1)

results = search.fit(xTrain, yTrain)

# save the model to disk
pickle.dump(results, open(data_name, 'wb'))

#print stats
cv = cross_validate(model, xTrain, yTrain, cv=5)
open_output = open(output_file, 'a')
open_output.write(data_name + '\n')
open_output.write('cross-validation:' + '\n')
open_output.write(str(cv) + '\n')
open_output.write('cross validation test score:' + '\n')
open_output.write(str(cv['test_score']) + '\n')
open_output.write('average cross validation score:' + str(cv['test_score'].mean()) + '\n')


The output is:

cross-validation:
{'fit_time': array([0.00080228, 0.00060129, 0.00053048, 0.00049925, 0.00050664]), 'score_time': array([0.00061917, 0.00049257, 0.00047731, 0.00050187, 0.00049067]), 'test_score': array([ -0.9680358 ,  -1.57469929,  -0.15547161,  -1.50097462,
-10.62839612])}

cross validation test score:
[ -0.9680358   -1.57469929  -0.15547161  -1.50097462 -10.62839612]

average cross validation score:-2.965515489454218


I'm really struggling to understand if this is a good score or not, and everywhere i read (including SO) is just that it's problem specific, hard to interpret, depends etc.

Can someone explain to me how do I progress this further, to understand how well my model worked on test data?

• Can you please explain what is unclear to you and what have you tried so far? – usεr11852 Mar 18 at 1:05
• Sure, what's unclear to me, is do these numbers demonstrate over-fitting or under-fitting, or do they represent a reasonably fit model (and so I can trust predictions). This is as far as I've tried with coding (because all that's left is to interpret), so I've been trying to read e.g. this and others, but I am not getting a clear answer on 'is the average cross-validation score of -2.9655 an indication of a reasonably fit model'? Thanks. – Ioannes Mar 18 at 1:52
• You make yourself no favours by not starting small and building up. Forget CV-grid-searches for a moment and try to work with a simple training-validation-test split first. Then move to more elaborate validation schema. Also, context matters: -2.96... might amazing or might be garbage in terms of goodness of fit depending on the application. – usεr11852 Mar 18 at 14:44
• great thanks, so is a higher score in the above example a 'better model' (better able to predict labels)? So e.g. if I have one model built with an average cv score of -10.3 and another model built with an average cv score of -2.3; are we saying that the -2.3 model is better able to predict labels? (And I shouldn't worry too much about what the actual number means itself)? – Ioannes Mar 18 at 15:25
• Generally speaking yes, -10.3 is worse than -2.3 because it is an RMSE. Please note that this bring us back to my earlier comment. Start small and build up; you being unable to readily interpreter your goodness of fit criteria shouts out that you have not done basic ground-work. Also for regression usually we refer to "response" instead of "label". Finally we should definitely worry about the metric, it relates to our problem. -2.3 does not mean anything by itself, just it "less bad" than "-10.3" but does not mean "-2.3 is good". – usεr11852 Mar 18 at 16:23