1
$\begingroup$

I'm making a cube learn to jump a jumprope by itself, and I don't know what approach would evolve to the most fitness, the fastest.

One approach could be setting up a basic feed-forward ANN with 1 hidden layer only, and tweaking the weights with a genetic algorithm.

On the other hand, another possibility would be to implement NEAT (genetic algorithm can change the number of connection and nodes, as well as the weights in the network).

Since the second approach is obviously more complicated, would it take more time to evolve the neural network than the one with a fixed structure?

Also, does it apply to all cases? For instance, I believe that for a basic jump rope, the basic neural network would do just fine.

$\endgroup$
1
$\begingroup$

genetic approaches are generally fairly junky. At least, since we have the gradient available, typically, using the gradient will learn faster.

It's true that Karpathy/OpenAI have proposed to use ES ("evolutionary strategies"), and showed that it learns faster, in terms of elapsed time, but they are parallelizing onto zillions of machines at a time. On a single machine, ES runs something like ten times slower than eg PG ("Policy gradients", a gradient approach), or perhaps TRPO ("trust region policy optimization"). Just with eg 100 computers in parallel, the ability to parallelize onto 100 machines, means that overall, ES will train 10 times faster than TRPO, albeit at huge expense, in terms of hardware cost.

NEAT only works for simple-tastic tasks. Look at the paper. They detect a big circle, in a picture with a big circle and a small circle.

If you must use genetic methods, ES is probably as good approach as any, since you have the resources/documentation of OpenAI behind it, eg https://blog.openai.com/evolution-strategies/

Edit: in response to questions:

1. why does neat only work for simple tasks?

Not using a gradient makes it very hard to know in which direction to move, in parameter hyperspace. NEAT doesnt handle functions that are not very smooth, eg "Critical Factors in the Performance of HyperNEAT", van der Berg and Whiteson, 2013, does some analysis on this point, using the term 'fracture' to describe this concept. (Admittedly this is for HyperNEAT, rather than basic NEAT). http://www.cs.ox.ac.uk/people/shimon.whiteson/pubs/vandenberggecco13.pdf

2. How to backprop in RL?

You run an episode, and get a reward. The reward is then the supervised training signal, that you backprop through the network, given the sequence of actions and states that you walked through during the episode as the inputs.

Highly recommend David Silver's lectures on the topic https://www.youtube.com/watch?v=2pWv7GOvuf0&list=PL7-jPKtc4r78-wCZcQn5IqyuWhBZ8fOxT

Alternatively, Karpathy's blog post on Policy Gradients to play Pong is pretty short and easy to read/understand, http://karpathy.github.io/2016/05/31/rl/

$\endgroup$
  • $\begingroup$ Very useful. I don't understand a couple of things, however: 1. I don't really understand why NEAT only works for simple tasks. Doesn't it allow to achieve more complex things? Encouraging the evolution and change of neural nets structures, and 2. I'm fairly new to machine learning, so I believe that in order to use a gradient, you need to give the network training data, right? I always link gradient to backpropagation, but I don't know if it's always the case. So, for my problem, I would only be able to use genetic approaches / ES. Is that right? $\endgroup$ – Nico de Ory Oct 22 '17 at 15:07
  • $\begingroup$ updated answer with answers to these questions $\endgroup$ – Hugh Perkins Oct 22 '17 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.