0
$\begingroup$

I wrote this code to run a SVR with cross validation:

def run_SVR(xTrain,yTrain,xTest,yTest,output_file,data_name):
  '''
  run SVR algorithm
  '''
  
  cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)

  # define the pipeline to evaluate
  model = SVR()
  fs = SelectKBest(score_func=mutual_info_regression)
  pipeline = Pipeline(steps=[('sel',fs), ('svr', model)])

  # define the grid
  grid = dict()
  grid['sel__k'] = [i for i in range(1, xTrain.shape[1]+1)]
  search = GridSearchCV(
        pipeline,
        param_grid={
            'svr__C': [0.01, 0.1, 1, 10, 100, 1000], ##Regularization
            'svr__epsilon': [0.0001, 0.001, 0.01, 0.1, 1, 10],
            'svr__gamma': [0.0001,  0.001, 0.01, 0.1, 1, 10],
        },
        scoring='neg_mean_squared_error',
        return_train_score=True,
        verbose=1,
        cv=5,
        n_jobs=-1)

  results = search.fit(xTrain, yTrain)

  # save the model to disk
  pickle.dump(results, open(data_name, 'wb'))

  #print stats
  cv = cross_validate(model, xTrain, yTrain, cv=5)
  open_output = open(output_file, 'a')
  open_output.write(data_name + '\n')
  open_output.write('cross-validation:' + '\n')
  open_output.write(str(cv) + '\n')
  open_output.write('cross validation test score:' + '\n')
  open_output.write(str(cv['test_score']) + '\n')
  open_output.write('average cross validation score:' + str(cv['test_score'].mean()) + '\n')

The output is:

cross-validation:
{'fit_time': array([0.00080228, 0.00060129, 0.00053048, 0.00049925, 0.00050664]), 'score_time': array([0.00061917, 0.00049257, 0.00047731, 0.00050187, 0.00049067]), 'test_score': array([ -0.9680358 ,  -1.57469929,  -0.15547161,  -1.50097462,
       -10.62839612])}

cross validation test score:
[ -0.9680358   -1.57469929  -0.15547161  -1.50097462 -10.62839612]

average cross validation score:-2.965515489454218

I'm really struggling to understand if this is a good score or not, and everywhere i read (including SO) is just that it's problem specific, hard to interpret, depends etc.

Can someone explain to me how do I progress this further, to understand how well my model worked on test data?

$\endgroup$
5
  • $\begingroup$ Can you please explain what is unclear to you and what have you tried so far? $\endgroup$ – usεr11852 Mar 18 at 1:05
  • $\begingroup$ Sure, what's unclear to me, is do these numbers demonstrate over-fitting or under-fitting, or do they represent a reasonably fit model (and so I can trust predictions). This is as far as I've tried with coding (because all that's left is to interpret), so I've been trying to read e.g. this and others, but I am not getting a clear answer on 'is the average cross-validation score of -2.9655 an indication of a reasonably fit model'? Thanks. $\endgroup$ – Ioannes Mar 18 at 1:52
  • $\begingroup$ You make yourself no favours by not starting small and building up. Forget CV-grid-searches for a moment and try to work with a simple training-validation-test split first. Then move to more elaborate validation schema. Also, context matters: -2.96... might amazing or might be garbage in terms of goodness of fit depending on the application. $\endgroup$ – usεr11852 Mar 18 at 14:44
  • $\begingroup$ great thanks, so is a higher score in the above example a 'better model' (better able to predict labels)? So e.g. if I have one model built with an average cv score of -10.3 and another model built with an average cv score of -2.3; are we saying that the -2.3 model is better able to predict labels? (And I shouldn't worry too much about what the actual number means itself)? $\endgroup$ – Ioannes Mar 18 at 15:25
  • $\begingroup$ Generally speaking yes, -10.3 is worse than -2.3 because it is an RMSE. Please note that this bring us back to my earlier comment. Start small and build up; you being unable to readily interpreter your goodness of fit criteria shouts out that you have not done basic ground-work. Also for regression usually we refer to "response" instead of "label". Finally we should definitely worry about the metric, it relates to our problem. -2.3 does not mean anything by itself, just it "less bad" than "-10.3" but does not mean "-2.3 is good". $\endgroup$ – usεr11852 Mar 18 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.