# Is initializing the weights of autoencoders still a difficult problem?

I was wondering if initializing the weights of autoencoders is still difficult and what the most recent strategies are for it.

I have been reading different articles. In one of Hinton's papers (2006), it says:

With large initial weights, autoencoders typically find poor local minima; with small initial weights, the gradients in the early layers are tiny, making it infeasible to train autoencoders with many hidden layers. If the initial weights are close to a good solution, gradient descent works well, but finding such initial weights requires a very different type of algorithm that learns one layer of features at a time. We introduce this "pretraining" procedure for binary data, generalize it to real-valued data, and show that it works well for a variety of data sets.

• no. any standard init (glorot / xavier / he) should work fine. – shimao Nov 11 '19 at 8:31

• The switch to non-saturating activation functions, like $$\operatorname{LReLU}_{0.1}(x) = \begin{cases}x & x \ge 0 \\ 0.1 \, x & x < 0\end{cases}$$, rather than saturating ones like $$\operatorname{sigmoid}(x) = \frac{1}{1 + \exp(-x)} \in (0, 1)$$ that were previously popular. Sigmoids only have useful signal for $$x$$ in a fairly tight set of inputs; too large or too small and the function becomes quite flat. Leaky ReLU has useful signal everywhere, and regular ReLU has useful signal for any positive input.