1
$\begingroup$

I am currently taking a class in machine learning. I had mentioned to a coworker that we were learning about randomized optimization, specifically randomized hill climbing (RHC). He said that it was possible to use RHC instead of backpropagation to find good weights for a neural network. Unfortunately, we didn't get to finish the conversation, and it's been bugging me all weekend. Does anyone know what he meant by that?

I've been playing around with Weka, using the weka.classifiers.bayes.net.search.local.K2 classifier...but I'm just not seeing how that would give me weights that could be used by a neural network, such as the multilayer perceptron.

Improve this question
$\endgroup$
5
  • $\begingroup$ Hill Climbing doesn't use gradient information, while Backprop does. Thus backprop works / converges much faster. $\endgroup$
    – Artem Sobolev
    Feb 9 '15 at 11:33
  • $\begingroup$ I am not an ANN expert, but a quick Google search reveals that the RHC algorithm involves changing a single weight, and see if the performance increases, and if so, keep this new weight. So it's a greedy method. You start with an initial set of weights, then changing one weight at a time, with a random amount (the maximum absolute value of this change might be called "learning rate" in this context) and if the performance of your classifier/regressor/whatever is better, than you will keep this new set of weights, and repeat until "convergence". $\endgroup$
    – jeff
    Sep 12 '15 at 2:29
  • $\begingroup$ Related to using RHC for ANN training is Neuroevolution, which uses evolutionary/genetic algorithms for determining the weights of a neural network. The core idea is the same: they determine if candidate (=weights) get better by their fitness (=NN error/performance). GAs with Neuroevolution might be good complementary information for understanding how such concepts can be used with ANN training. $\endgroup$
    – geekoverdose
    Jun 15 '16 at 13:15
  • $\begingroup$ You may also be interested in a few other questions about alternate ways to train a neural network, like stats.stackexchange.com/questions/207450/… and stats.stackexchange.com/questions/235862/… $\endgroup$
    – Matt Krause
    Aug 23 '18 at 1:10
  • $\begingroup$ For future reference, Genevieve Hayes has done a lot of work on this subject, and has developed a python library, mlrose, to help facilitate. $\endgroup$
    – Mike Dunn
    Feb 26 '20 at 19:55
1
$\begingroup$

Recent papers show that there is nearly always some small step you can take to head toward a good solution: Saddle point problem It seems to me that something like random hill climbing (RHC) is fine for that. In practice I am starting to see that in my code too. For deep neural networks I wouldn't be surprised if RHC and back-propagation were comparable in the amount of computation required.

Improve this answer
$\endgroup$
-1
$\begingroup$

Random search has great probability of finding you a solution among the top ones, see my answer to Practical hyperparameter optimization: Random vs. grid search.

So, with only 60 tries, and a constrained parameter set, you have 95 percent probability of finding a solution among the top 5 percent best solutions (inside those constraints). Great, right? Well...

Actually, it turns out that solution might still be totally garbage, and you might only get good solutions from NNs into the 0.1 or even less percent of solution, requiring even more trials to have a good probability of finding.

Improve this answer
$\endgroup$
6
  • $\begingroup$ The question seems to ask about training the NN rather then finding hyperparameters. $\endgroup$
    – Tim ♦
    Feb 26 '20 at 20:05
  • $\begingroup$ @Tim it amounts to the same thing in this case $\endgroup$
    – Firebug
    Feb 26 '20 at 21:01
  • $\begingroup$ Not really. Algorithm that works for few parameters may not be efficient for few million parameters. Pure random search works also quite good for hyperparameter tuning, try it for training the model. $\endgroup$
    – Tim ♦
    Feb 26 '20 at 21:43
  • $\begingroup$ @Tim I say exactly that on the answer, so we... agree? I don't see your point tbh. $\endgroup$
    – Firebug
    Feb 27 '20 at 14:45
  • $\begingroup$ "might be garbage" does not sound like "it is unlikely to work at all". $\endgroup$
    – Tim ♦
    Feb 27 '20 at 15:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.