Best Way to do Hyperparam Search and Cross-Validation

Question

I'm making experiments to evaluate language models to brazilian portuguese datasets.

So, i've made so each dataset is divided in 10 parts, I want to use cross-validation to determine the model's performance. But the thing is, I also want to use Hyperparam Search to determine the best parameter for the model.

I've red that you should do Hyperparam Search for each fold of the cross validation, and in other places, that you have to use a part of the dataset that will not be used in the cross validation.

Can someone please give me a hand on this one? I would also appreciate if someone has some academical articles that describe the process please :)

Answer 1

So I don't know if I got the idea right. But if you test it explicitly, you'll have to break some functions. I did a quick search in the sk-learn library, I'll leave some links.

grid_search

cross_validation

cross_validate

KFold

GridSearchCV

RandomizedSearchCV

If I understand the idea of passing the Hyperparam to each node that cross validation is going to pass. First we have to see how the cross-validation behaves, these links have very good explanations.

In the code below I put a function similar to cross validation without hyperparam:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn import datasets
from sklearn import svm
from sklearn.model_selection import KFold

X, y = datasets.load_iris(return_X_y=True)


kf = KFold(n_splits=2)
for i, (X_train_index_fold, X_test_index_fold) in enumerate(kf.split(X)):
    print(f"Fold {i}:")
    print(f'Shape {X.shape}')
    print(f"  Train: shape={X_train_index_fold.shape}")
    print(f"  Test:  shape={X_test_index_fold.shape}")
    """
    kf.split will separate by different index, then I just 
     add the indexes that were selected in my numpy array, 
     which it returns to me.
    """
    X_train, X_test, y_train, y_test = train_test_split(
    X[X_train_index_fold], y[X_test_index_fold], test_size=0.3, random_state=0)
    print(f" X train: shape={X_train.shape}")
    print(f" X test: shape={X_test.shape}")
    print(f" y train:  shape={y_train.shape}")
    print(f" y test: shape={y_test.shape}")
    clf = svm.SVC(kernel='linear', C=1).fit(X_train, y_train)
    print(f'Fold {i} | Score: {clf.score(X_test, y_test)}')

I recommend that you print the steps, you can see the processes individually.

output

Fold 0:
Shape (150, 4)
  Train: shape=(75,)
  Test:  shape=(75,)
 X train: shape=(52, 4)
 X test: shape=(23, 4)
 y train:  shape=(52,)
 y test: shape=(23,)
Fold 0 | Score: 0.5217391304347826
Fold 1:
Shape (150, 4)
  Train: shape=(75,)
  Test:  shape=(75,)
 X train: shape=(52, 4)
 X test: shape=(23, 4)
 y train:  shape=(52,)
 y test: shape=(23,)
Fold 1 | Score: 0.6086956521739131

You can see that I can separate each node, with different index to be able to train the model again, passing through the classifier

This very simple example how the nodes are separated.

At the point I couldn't understand, you want to pass the hyperparam, because the link I left above GridSearch and RandomizedSearchCV. You already pass the cross validation function as a parameter, well if you have a very large set you will be able to pass it several times, but if it is too short it will return an error. Being a very populated set it will run twice, using a similar function from above, and then passing the hyperparam again. If you give more details edited this answer, to see if within my knowledge I can help you.

GridSearch documentation info:

cv int, cross-validation generator or an iterable, default=None
Determines the cross-validation splitting strategy. Possible inputs for cv are:

None, to use the default 5-fold cross validation,

integer, to specify the number of folds in a (Stratified)KFold,

CV splitter,

An iterable yielding (train, test) splits as arrays of indices.

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used. These splitters are instantiated with shuffle=False so the splits will be the same across calls.

Refer User Guide for the various cross-validation strategies that can be used here.

Tamo junto bro, brazuca na área. 🇧🇷