0
$\begingroup$

I realise that nested cross validation can be used to reduce bias when hyper-parameters tuning is combined with model selection. However, I wonder if it is possible to perform hyper-parameter tuning or feature selection as a separate step using grid search cv on the entire training dataset (The entire dataset is split into training and test set). This would identify the best set of parameters for the training dataset (see code below).

https://machinelearningmastery.com/nested-cross-validation-for-machine-learning-with-python/

The best hyper-parameter or features can then be used for subsequent cross validation on the a newly instantiated model with the optimal hyper-parameters or features identified in the previous step.

I wonder if this would be preferable to nested cross validation given the scenario that finding the best set of hyperparameter and features is important. Since the model is re-instantiated during the second cross validation (for model selection), there should no over-fitting.

The test set is not considered here, as once model selection is complete, the best model(s) can then be applied to the test set.

I wonder if this approach is reasonable? Any help is appreciated.

from sklearn.model_selection import GridSearchCV
from sklearn import svm
from sklearn.svm import LinearSVC
import numpy as np

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

SVCpipe = Pipeline([('scale', StandardScaler()),
                   ('SVC',LinearSVC(class_weight = 'balanced',  dual=False, random_state = np.random.seed(7), tol=1e-05))])  #Prefer dual=False when n_samples > n_features.

param_grid = {'SVC__C':np.arange(0.01,100,10)}

X_train = cardioDataCal_Eurosc_II_Train.iloc[:, 0:18]

linearSVC = GridSearchCV(SVCpipe, param_grid,cv=3,return_train_score=True, refit=True,verbose=2, scoring='f1') #https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter
%time linearSVC.fit(X_train, cardioDataCal_Eurosc_II_TrainLabel.values.ravel())
#print(linearSVC.best_estimator_)

print(linearSVC.best_estimator_)

Cross validation of entire training dataset using 
best hyperparameters and features identified in the above section.

```
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.