0
$\begingroup$

I am working on a school project using remote sensing data, for classification purposes. And I am trying to select the best model (models) for my data. The approach that I adopted is the following:

  1. Looping over different models and using K-fold cross-validation (k=10).
  2. Select the best performing model (models)
  3. Parameter grid search (or random search) for each of the models I chose in (2)
  4. Cross-validation using the best parameters I found in (3)
  5. Perform prediction

Does this approach make sense? I am still in step 1 and it is already taking up so much time. I am new to machine learning, so I wanted to get some insight into how people in this domain usually tackle projects. Any advice would be highly appreciated.

$\endgroup$
1
0
$\begingroup$

You should optimize the hyperparameters of each class of model first and then compare the different classes of model to each other. Otherwise, the performance of the different model types will be heavily influenced by the arbitrary selection of hyperparameters. For example, with the initial parameters, it may be the case that a random forest performs better than SVM. However, once both have the best hyperparameters, it may be the reverse case. If you had chosen the random forest and then optimized, you'd end up with a worse than optimal model because you never even tested the optimal SVM model.

Using loops where you can and reusing old code will speed things up. Also, depending on your program, there are functions which implement CV and hyperparameter selection, so you don't need to code that by hand. In Python you can use GridSearchCV.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for your response. I was actually debating whether I should perform hyperparameter tuning first and then choose the best model or do the opposite. Thanks for clarifying this! $\endgroup$ – Rim Sleimi Jun 2 '20 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.