I use NN for my mini project research, and I found out the newest trick for feed forward NN is using dropout for regularization instead of L1/L2 norm and rectified linear unit as an activation function.
But when I tried it, I always got worse results compared to a standard NN with sigmoid / hyperbolic tangent activation function.
Is there some rule of thumb or trick that we can use for training dropout ReLU NN?