For every N model:
Split in test and train subsets(Using the same seed for every N model)
Randomized Search of parameters with 5 k-folds on train subset
Select the best estimator obtained after the Randomized Search based on accuracy.
Test the tuned model with test subset and get accuracy score
Select the model with the best accuracy score obtained in 4.
Train the selected model with all data
Is this right for model selection?
I'm a bit confused because I've read that we need to split in three subsets: test, validation and train. How different is this approach compared with my current approach?
Python Code for only one model, I do this for N models then select best score
pipeline = Pipeline([ ('union', FeatureUnion( transformer_list=[ ('ordinal', Pipeline([ ('selector', ordinalSelector), ('Imputer', preprocessing.Imputer(-999, strategy='mean')), ])), ('nominal', Pipeline([ ('selector', nominalSelector), ('Imputer', preprocessing.Imputer(-999, strategy='most_frequent')), ('OneHot', preprocessing.OneHotEncoder(sparse=False)), ])), ], )), ('MinMaxScaler',preprocessing.MinMaxScaler([-1,1])), ('SGD', SGDClassifier(class_weight='balanced', shuffle=True))]) X_train, X_test, y_train, y_test = train_test_split(X,y ,test_size=0.50) cv=StratifiedKFold(y_train, n_folds=5, shuffle=True) grid = RandomizedSearchCV(pipeline, param_distributions=param_dist, scoring='roc_auc', cv=cv) random_search.fit(X_train, y_train) y_pred = random_search.predict(X_test) print(classification_report(y_test, y_pred, target_names=classes_names))