1
$\begingroup$

I am currently taking a class in machine learning. I had mentioned to a coworker that we were learning about randomized optimization, specifically randomized hill climbing (RHC). He said that it was possible to use RHC instead of backpropagation to find good weights for a neural network. Unfortunately, we didn't get to finish the conversation, and it's been bugging me all weekend. Does anyone know what he meant by that?

I've been playing around with Weka, using the weka.classifiers.bayes.net.search.local.K2 classifier...but I'm just not seeing how that would give me weights that could be used by a neural network, such as the multilayer perceptron.

Improve this question
$\endgroup$
5
  • $\begingroup$ Hill Climbing doesn't use gradient information, while Backprop does. Thus backprop works / converges much faster. $\endgroup$
    – Artem Sobolev
    Commented Feb 9, 2015 at 11:33
  • $\begingroup$ I am not an ANN expert, but a quick Google search reveals that the RHC algorithm involves changing a single weight, and see if the performance increases, and if so, keep this new weight. So it's a greedy method. You start with an initial set of weights, then changing one weight at a time, with a random amount (the maximum absolute value of this change might be called "learning rate" in this context) and if the performance of your classifier/regressor/whatever is better, than you will keep this new set of weights, and repeat until "convergence". $\endgroup$
    – jeff
    Commented Sep 12, 2015 at 2:29
  • $\begingroup$ Related to using RHC for ANN training is Neuroevolution, which uses evolutionary/genetic algorithms for determining the weights of a neural network. The core idea is the same: they determine if candidate (=weights) get better by their fitness (=NN error/performance). GAs with Neuroevolution might be good complementary information for understanding how such concepts can be used with ANN training. $\endgroup$
    – geekoverdose
    Commented Jun 15, 2016 at 13:15
  • $\begingroup$ You may also be interested in a few other questions about alternate ways to train a neural network, like stats.stackexchange.com/questions/207450/… and stats.stackexchange.com/questions/235862/… $\endgroup$
    – Matt Krause
    Commented Aug 23, 2018 at 1:10
  • $\begingroup$ For future reference, Genevieve Hayes has done a lot of work on this subject, and has developed a python library, mlrose, to help facilitate. $\endgroup$
    – Mike Dunn
    Commented Feb 26, 2020 at 19:55

2 Answers 2

Reset to default
1
$\begingroup$

Recent papers show that there is nearly always some small step you can take to head toward a good solution: Saddle point problem It seems to me that something like random hill climbing (RHC) is fine for that. In practice I am starting to see that in my code too. For deep neural networks I wouldn't be surprised if RHC and back-propagation were comparable in the amount of computation required.

Improve this answer
$\endgroup$
-1
$\begingroup$

Random search has great probability of finding you a solution among the top ones, see my answer to Practical hyperparameter optimization: Random vs. grid search.

So, with only 60 tries, and a constrained parameter set, you have 95 percent probability of finding a solution among the top 5 percent best solutions (inside those constraints). Great, right? Well...

Actually, it turns out that solution might still be totally garbage, and you might only get good solutions from NNs into the 0.1 or even less percent of solution, requiring even more trials to have a good probability of finding.

Improve this answer
$\endgroup$
6
  • $\begingroup$ The question seems to ask about training the NN rather then finding hyperparameters. $\endgroup$
    – Tim
    Commented Feb 26, 2020 at 20:05
  • $\begingroup$ @Tim it amounts to the same thing in this case $\endgroup$
    – Firebug
    Commented Feb 26, 2020 at 21:01
  • $\begingroup$ Not really. Algorithm that works for few parameters may not be efficient for few million parameters. Pure random search works also quite good for hyperparameter tuning, try it for training the model. $\endgroup$
    – Tim
    Commented Feb 26, 2020 at 21:43
  • $\begingroup$ @Tim I say exactly that on the answer, so we... agree? I don't see your point tbh. $\endgroup$
    – Firebug
    Commented Feb 27, 2020 at 14:45
  • $\begingroup$ "might be garbage" does not sound like "it is unlikely to work at all". $\endgroup$
    – Tim
    Commented Feb 27, 2020 at 15:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.