0
$\begingroup$

Suppose you are performing a big hyper-parameter search, using scikit learn RandomizedSearchCV or GridSearchCV. Suppose you are running it on some shared platform, say Google Colab, that disconnects you after a while, so that it is impossible to perform the entire search all together.

It would be useful to be able to save the state of the search in the disk, in order to be able to resume the search from the same point some time later.

I know one can achieve this "manually", but not exploring all the parameter space at once, but dividing it in different regions and run the search on each region separately.

However, I am looking for something that does not require any manual intervention or any additional code to subdivide the parameter space.

I think such functionality may be available in other libraries like skopt.

Do you know if what I am asking for is possible in RandomizedSearchCV and GridSearchCV from sklearn?

$\endgroup$
0
$\begingroup$

You can run RandomizedSearchCV multiple times with different seeds and check if the score is better than the best you found previously. The code would look like

if not os.path.exists('res.pkl'):
    best_score, seed = -1, -1  # initialize
else:
    with open('res.pkl', 'r') as fd:
        best_score, _, seed = pickle.load(fd)  # or get the best score so far and the last seed tried
while True:
    seed += 1
    search = RandomizedSearchCV(..., random_state=seed)
    search.fit(...)  # try some more random parameters
    if search.best_score_ > best_score:  # if it beats the best score so far
        best_score = search.best_score_
        with open('res.pkl', 'r') as fd:
            pickle.load((best_score, search.best_params_, seed), fd)    # store the new best parameters

It would be harder to code that for GridSearchCV as you need to partition the grid and keep track of what you tried so far. RandomizedSearchCV is probably the way to go though if you have more than very few parameters to optimize.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.