For every N model:

  1. Split in test and train subsets(Using the same seed for every N model)

  2. Randomized Search of parameters with 5 k-folds on train subset

  3. Select the best estimator obtained after the Randomized Search based on accuracy.

  4. Test the tuned model with test subset and get accuracy score


  1. Select the model with the best accuracy score obtained in 4.

  2. Train the selected model with all data

Is this right for model selection?

I'm a bit confused because I've read that we need to split in three subsets: test, validation and train. How different is this approach compared with my current approach?


Python Code for only one model, I do this for N models then select best score

pipeline = Pipeline([
('union', FeatureUnion(
        ('ordinal', Pipeline([
            ('selector', ordinalSelector),
            ('Imputer', preprocessing.Imputer(-999, strategy='mean')),
        ('nominal', Pipeline([
            ('selector', nominalSelector),
            ('Imputer', preprocessing.Imputer(-999, strategy='most_frequent')),
            ('OneHot', preprocessing.OneHotEncoder(sparse=False)),
('SGD', SGDClassifier(class_weight='balanced', shuffle=True))])
X_train, X_test, y_train, y_test = train_test_split(X,y ,test_size=0.50)  
cv=StratifiedKFold(y_train, n_folds=5, shuffle=True)
grid = RandomizedSearchCV(pipeline, param_distributions=param_dist,   scoring='roc_auc', cv=cv)
random_search.fit(X_train, y_train)
y_pred = random_search.predict(X_test)
print(classification_report(y_test, y_pred, target_names=classes_names))

Hyperparameter optimization is a part of model selection, since you want to determine both the best learning method and its hyperparameterization.

The right way to deal with this is via nested cross-validation, which looks something like this:

1. cross-validate(data)
    --> train_outer and test_outer

    1.1 cross-validate(train_outer)
        --> train_inner and test_inner

        evaluate performance of given model for each fold
        where model includes learning method & hyperparameterization

        1.1.1 train model on train_inner (with given hyperparameterization)
        1.1.2 test trained model on test_inner
        1.1.3 compute performance metric of your choice

    1.2 select best modeling approach and train on train_outer
    (the best is determined based on performance in (1.1))

    1.3 estimate performance of best approach from 1.1 on test_outer

2. aggregate results from cross-validation (1) for performance estimate

The result of a nested cross-validation procedure is a fair performance estimate of your whole modelling pipeline, which includes hyperparameter optimization and, optionally, learning algorithm selection.

Finally, you mention randomized search to find suitable hyperparameters. It's worth noting that much better methods exist, in libraries such as Optunity. These dedicated methods will give you better sets of hyperparameters than random search given the same budget.

To see an example of this in Python, please refer to the nested cross-validation example in Optunity's documentation. You can expand the same setup to include learning algorithm selection instead of limiting yourself to optimizing the hyperparameterization for a predefined approach.

  • $\begingroup$ So if I have two different models(i.e. Neural Network and RBF kernel SVM) I have to compare and select between them on inner cv(1.2)? then, why I have to do nested cv? $\endgroup$ Oct 12 '15 at 5:39
  • $\begingroup$ I think I'm understanding now, but what's happens if I have unbalanced data (250000 and 7500) it's the same method? $\endgroup$ Oct 12 '15 at 6:09
  • $\begingroup$ Yes, you select based on the results of inner cross-validation. If you have unbalanced data, everything remains essentially the same, except that you might want to stratify the generation of folds in both cross-validations to ensure that the class balance is equal across all folds. $\endgroup$ Oct 12 '15 at 7:19
  • $\begingroup$ I'm reading optunity docs, and looked at optunity.readthedocs.org/en/latest/api/… seems that I need to use strata param for doing a stratify the generation of folds, right? Optunity works with pandas? $\endgroup$ Oct 13 '15 at 2:57
  • $\begingroup$ @FelipeElíasLolasIsuani yes, that's right, you can use strata=optunity.cross_validation.strata_by_labels(). Optunity is fully compatible with numpy. We didn't test integration with pandas very much but it shouldn't pose too many problems, if any. $\endgroup$ Oct 13 '15 at 4:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.