0
$\begingroup$

I am relatively new to machine learning and am trying to implement an SVM for the first time on a project, but I'm running into some overfitting-related issues.

Basically, I created a function called optimize() to optimize the hyperparameters C and gamma (using an rbf kernel) based on the average cross-validation accuracy obtained with each combination of gamma and C within a grid search (10^-3 to 10^3). I tested optimize() on the Iris dataset, using a 75%-25% training-testing split. I ran optimize() on the training set, and trained the model on this set using the hyperparameters that gave the best accuracy. I end up with a really high accuracy on the training set (~97%), but when I apply it to the test set I end up with really low accuracy (~32%).

I know my problems most likely have to do with overfitting on the training set, but that confuses me since I thought that using cross-validation to tune the hyperparameters would avoid that. Any suggestions would be appreciated, thanks :)

As suggested, I've added the function in question, written in Python, below.

NOTE: normalize(x_train, x_test) normalizes the train and test data by subtracting the mean and dividing by the standard deviation of each feature in the train set.

def optimize(X, Y, fold=3):
    acc_list = []
    print("")
    print("Initiating Grid Search ...")
    print("")
    for i in range(-3, 4):
        print("C = {}".format(10**i))
        sub_acc_list = []
        for j in range(-3, 4):
            print("     Gamma = {}".format(10**j))
            start = 0
            stop = 0
            validate_list = []
            for i in range(fold):
                width = int(0.25*len(X))
                start = random.randint(0, (len(X)-1)-width)
                stop = start + width

                train_features = X[:start] + X[stop:]
                test_features = X[start:stop]
                train_features, test_features = normalize(train_features, test_features)
                train_labels = Y[:start] + Y[stop:]
                test_labels = Y[start:stop]

                model = svm.SVC(kernel="rbf", gamma=10**j, C=10**i, decision_function_shape="ovo")
                model.fit(train_features, train_labels)
                output = model.predict(test_features)

                num_correct = 0
                for k in range(len(output)):
                    if output[k] == test_labels[k]:
                        num_correct += 1
                        # print(num_correct)
                validate_list += [(num_correct/float(len(output)))*100, ]
                # print("             Validate List = {}".format(validate_list))
            sub_acc_list += [round(average(validate_list), 0), ]
        acc_list += [sub_acc_list, ]
    max_acc = 0
    max_parameters = ()
    for i in range(len(acc_list)):
        for j in range(len(acc_list[i])):
            if acc_list[i][j] > max_acc:
                max_parameters = (10**i, 10**j)
                max_acc = acc_list[i][j]
    print("")
    print("Best Accuracy: {}%".format(max_acc))
    print("")
    model = svm.SVC(C=max_parameters[0], gamma=max_parameters[1], decision_function_shape="ovo")
    model.fit(X, Y)
    return model
$\endgroup$
  • 1
    $\begingroup$ One possibility is that you have a non-random train-test split. Did you shuffle your data before the split? You can check which target values ended up in the train set and which ones ended up in the test set and see if the patterns are similar. $\endgroup$ – AlexK Apr 5 at 21:20
  • $\begingroup$ @AlexK Thanks a bunch for the suggestion. I did make sure to shuffle everything beforehand, and my train-test split is done by taking a random chunk of 40 or so examples from the shuffled dataset. After some more exploring I found that it seems to be predicting the same label for ALL the test examples, which makes sense why the accuracy is about 30% since there are 3 labels to pick from. $\endgroup$ – AlexP Apr 6 at 16:34
  • 1
    $\begingroup$ The accuracy should definitely not be that low. It seems like something is off in your training pipeline, so I recommend you add your code to the question. $\endgroup$ – AlexK Apr 6 at 20:06
  • $\begingroup$ Thanks, I've added the optimize() function to the post. $\endgroup$ – AlexP Apr 6 at 22:32
  • 1
    $\begingroup$ Don't appear to be any major issues with this code, other than you have two i counters in the nested for loops and that seems like an odd type of cross-validation where you allow the same samples to appear in different training iterations. You should add the remaining code, and probably post it on Stack Overflow. The answer to your original question is yes, cross-validation, at least in one of its standard forms like k-fold and stratified k-fold, is meant to make your training results generalizable to unseen data. $\endgroup$ – AlexK Apr 7 at 4:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.