I'm writing my own neural net from scratch (using Clojure).
For the activation function for the nodes, I'm using the Sigmoid Function.
I was messing around manually creating "trained" nets based on example logic gates from this page, and noticed that my AND gate wasn't giving me as nice of values back as I'd like. When giving it
1,1, it would give back a number around 0.6 instead of a hard 1, or at least 0.99999.
0,0 would give back a number around 0.3 instead of a tiny number near 0, and
1,0 gave back a number around 0.4. I realized I needed to make the graph of the Sigmoid function "steeper" to get more extreme values, so I started playing with the function on Wolfram Alpha.
I noticed that if I change the constant
e to something larger, it creates a much steeper graph than the original kind of lazy curve. When I changed the activation function to the modified steeper version, I did in fact get back nicer values.
Now, my for my question:
I decided to try training my net while using the steeper version, and found that you can get fairly significant training speedups by making the graph steeper; although you seem to get diminishing returns once you change from
e to a value larger than 20. Here are the errors after 1 million fires and backpropagation corrections of the network for different "scaling" values when training an AND gate:
It seems as though increasing the scale of the sigmoid function increases how fast the net is able to learn; even when using the same learning rate.
Is there a rule of thumb regarding what value to use to scale the Sigmoid function when using it as an activation function?