I'm implementing a deep denoising autoencoder using Theano in Python. I know that you are supposed to initialise starting weights using random numbers and that's what Theano authors do in their tutorials, though they use weird upper and lower bounds without giving any explanation. So, here is he code:

starting_weights = np.asarray(numpy_rng.uniform(
        low=-4 * np.sqrt(6. / (n_hidden + n_visible)),
        high=4 * np.sqrt(6. / (n_hidden + n_visible)),
        size=(n_visible, n_hidden)), dtype=theano.config.floatX

Basically, they take a sample from the uniform distribution truncated at $ \pm 4\cdot \sqrt { \frac { 6 }{ n+m } } $, where $m$ and $n$ are the numbers of visible and hidden units. Can someone explain what is the rationale behind it?


You can find a short note about this initialisation step here. Different activation functions should have different initialisation bounds. The more detailed explanation you can find in this paper in the 4.2.1 section. They don't use factor of 4, but I assume it's related to the nature of sigmoid and hyperbolic tangent function. They have very similar behaviour, but hyperbolic tangent reach its asymptotes faster. You can compare them in the plot:

x = np.linspace(-5, 5, 10000)

plt.figure(figsize=(12, 8))
plt.plot(x, np.tanh(x))
plt.plot(x, (1 / (1 + np.exp(-x))))

enter image description here

where the blue function is a sigmoid and the red one is a hyperbolic tangent.

Sigmoid approximates to its asymptote slower, therefore, it needs to have bigger values in weight matrix to get the same output as hyperbolic tangent.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.