2
$\begingroup$

I want to do a convergence study of Ridge regression with increasing training size. For this I train the Ridge regressor using hyperopt for different training sizes [10, 20, 40, 80]. As far as I understand, the TPE strategy adapts to the optimization, so I would expect that the sampling for each training with different training sizes be different, but that's not the case. Here is a code that shows that TPE is always sampling the same points:

import numpy as np
X = np.random.randn(100,1)
Y = np.random.randn(100,1)

from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y,
                                                    test_size=0.2,
                                                    random_state=0)

from sklearn.linear_model import Ridge
from hyperopt import fmin, tpe, hp, Trials

results = []
for n in [10, 20, 40, 80]:
    print "\nTraining with", n, "samples"

    def objective(alpha):
        model = Ridge(alpha=alpha).fit(X_train[:n], Y_train[:n])
        return -model.score(X_test, Y_test)  # negative so that lower means better

    trials = Trials()
    best = fmin(objective, hp.uniform('alpha', 0.0, 10.0),
                algo=tpe.suggest,
                max_evals=5,
                trials=trials)

    print "alphas sampled:"
    for trial in trials:
        print trial['misc']['vals']

The output of this is:

Training with 10 samples
alphas tried:
{'alpha': [6.964691855978616]}
{'alpha': [1.0606490595473272]}
{'alpha': [5.067260114995699]}
{'alpha': [1.0517212721767522]}
{'alpha': [5.238333199546759]}

Training with 20 samples
alphas tried:
{'alpha': [6.964691855978616]}
{'alpha': [1.0606490595473272]}
{'alpha': [5.067260114995699]}
{'alpha': [1.0517212721767522]}
{'alpha': [5.238333199546759]}

Training with 40 samples
alphas tried:
{'alpha': [6.964691855978616]}
{'alpha': [1.0606490595473272]}
{'alpha': [5.067260114995699]}
{'alpha': [1.0517212721767522]}
{'alpha': [5.238333199546759]}

Training with 80 samples
alphas tried:
{'alpha': [6.964691855978616]}
{'alpha': [1.0606490595473272]}
{'alpha': [5.067260114995699]}
{'alpha': [1.0517212721767522]}
{'alpha': [5.238333199546759]}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.