3
$\begingroup$

I have tried xgboost on MNIST dataset with default settings and using early stopping.

Why I get worser results with early stopping in terms of accuracy? (93.4% vs 92.8%)

Here is samples of code:

With early stopping:

def train():
    X, y= mnist_dataset.load_train_data()
    X_train, X_test, y_train, y_test= train_test_split(X, y, test_size=0.20, random_state=2017)

    print ('X_train.shape', X_train.shape)
    print ('y_train.shape', y_train.shape)
    print ('X_test.shape', X_test.shape)
    print ('y_test.shape', y_test.shape)

    xgb_train = xgb.DMatrix(X_train, label=y_train)
    xgb_test = xgb.DMatrix(X_test, label=y_test)

    clf = xgb.XGBClassifier(objective = 'multi:softmax',
                            silent = True,
                            nthread = multiprocessing.cpu_count()
                            )
    clf.fit(X_train, y_train,
        early_stopping_rounds=10, eval_metric="merror", eval_set=[(X_test, y_test)],
        verbose=True)

    y_pred = clf.predict(X_test)

    print('acc:',100.0*accuracy_score(y_test, y_pred)) 

With default settings:

def train():
    X, y= mnist_dataset.load_train_data()
    X_train, X_test, y_train, y_test= train_test_split(X, y, test_size=0.20, random_state=2017)

    print ('X_train.shape', X_train.shape)
    print ('y_train.shape', y_train.shape)
    print ('X_test.shape', X_test.shape)
    print ('y_test.shape', y_test.shape)

    xgb_train = xgb.DMatrix(X_train, label=y_train)
    xgb_test = xgb.DMatrix(X_test, label=y_test)

    # setup parameters for xgboost
    param = {}
    param['objective'] = 'multi:softmax'
    param['silent'] = 1
    param['nthread'] = multiprocessing.cpu_count()
    param['num_class'] = 10

    clf = xgb.train(param, xgb_train)

    y_pred = clf.predict(xgb_test)

    print('acc:',100.0*accuracy_score(y_test, y_pred))
$\endgroup$
  • 2
    $\begingroup$ how many rows/images are there in that data-set? Have you tried increasing the early-stopping rounds? Especially with small data sets early stopping usually leads to worse results. Check out this link which compares using early stopping vs not using it with GBTs. Early stopping at least makes sure your not running too many rounds and hopefully doesn't overfit. You are probably underfitting. $\endgroup$ – user3494047 Feb 1 '18 at 16:26
1
$\begingroup$

Your code is measuring out-of-sample cross-entropy loss to determine when to stop. Cross-entropy and accuracy don't measure the same thing, so it's not surprising that you get different results.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.