Can I do hyper-parameters optimization before model selection?

Question

For every N model:

1. Split in test and train subsets(Using the same seed for every N model)

2. Randomized Search of parameters with 5 k-folds on train subset

3. Select the best estimator obtained after the Randomized Search based on accuracy.

4. Test the tuned model with test subset and get accuracy score

Then:

1. Select the model with the best accuracy score obtained in 4.

2. Train the selected model with all data

---

Is this right for model selection?

I'm a bit confused because I've read that we need to split in three subsets: test, validation and train. How different is this approach compared with my current approach?

UPDATE

Python Code for only one model, I do this for N models then select best score

pipeline = Pipeline([
('union', FeatureUnion(
    transformer_list=[
        ('ordinal', Pipeline([
            ('selector', ordinalSelector),
            ('Imputer', preprocessing.Imputer(-999, strategy='mean')),
        ])),
        ('nominal', Pipeline([
            ('selector', nominalSelector),
            ('Imputer', preprocessing.Imputer(-999, strategy='most_frequent')),
            ('OneHot', preprocessing.OneHotEncoder(sparse=False)),
        ])),
    ],
)),
('MinMaxScaler',preprocessing.MinMaxScaler([-1,1])),
('SGD', SGDClassifier(class_weight='balanced', shuffle=True))])
X_train, X_test, y_train, y_test = train_test_split(X,y ,test_size=0.50)  
cv=StratifiedKFold(y_train, n_folds=5, shuffle=True)
grid = RandomizedSearchCV(pipeline, param_distributions=param_dist,   scoring='roc_auc', cv=cv)
random_search.fit(X_train, y_train)
y_pred = random_search.predict(X_test)
print(classification_report(y_test, y_pred, target_names=classes_names))

Answer 1

Hyperparameter optimization is a part of model selection, since you want to determine both the best learning method and its hyperparameterization.

The right way to deal with this is via nested cross-validation, which looks something like this:

1. cross-validate(data)
    --> train_outer and test_outer

    1.1 cross-validate(train_outer)
        --> train_inner and test_inner

        evaluate performance of given model for each fold
        where model includes learning method & hyperparameterization

        1.1.1 train model on train_inner (with given hyperparameterization)
        1.1.2 test trained model on test_inner
        1.1.3 compute performance metric of your choice

    1.2 select best modeling approach and train on train_outer
    (the best is determined based on performance in (1.1))

    1.3 estimate performance of best approach from 1.1 on test_outer

2. aggregate results from cross-validation (1) for performance estimate

The result of a nested cross-validation procedure is a fair performance estimate of your whole modelling pipeline, which includes hyperparameter optimization and, optionally, learning algorithm selection.

Finally, you mention randomized search to find suitable hyperparameters. It's worth noting that much better methods exist, in libraries such as Optunity. These dedicated methods will give you better sets of hyperparameters than random search given the same budget.

To see an example of this in Python, please refer to the nested cross-validation example in Optunity's documentation. You can expand the same setup to include learning algorithm selection instead of limiting yourself to optimizing the hyperparameterization for a predefined approach.