4
$\begingroup$

I use NN for my mini project research, and I found out the newest trick for feed forward NN is using dropout for regularization instead of L1/L2 norm and rectified linear unit as an activation function.

But when I tried it, I always got worse results compared to a standard NN with sigmoid / hyperbolic tangent activation function.

Is there some rule of thumb or trick that we can use for training dropout ReLU NN?

Improve this question
$\endgroup$
2
  • $\begingroup$ What do you mean by "always", i.e. how many parameter settings, how data sets have you checked? What do you mean by "worse results", prediction or training error? Finally, do not believe every statement you read in papers, they try to sell you their ideas because they have to. $\endgroup$
    – davidhigh
    Jul 9, 2014 at 10:28
  • $\begingroup$ From now I already tried it on several dataset which is public available like MNIST or my own dataset, all of them give worse result than standard MLP with sigmoid. I use same layer configuration but for ReLu I tried lower learning rate. So I just want some rule of thumb for nn with dropout & relu hyperparameter (like bigger or smaller hidden layer size, bigger or smaller learning rate) compared to standard sigmoid MLP $\endgroup$
    – psuedobot
    Jul 10, 2014 at 5:04

1 Answer 1

Reset to default
7
$\begingroup$

I am posting quite late, but I wanted to provide an answer just in case someone else has this problem.

Check that you are turning off dropout when you are evaluating on the validation/test set or if you want to compute error on the training set. Dropout was designed with the express intent of reducing overfitting, so if you are evaluating training loss with dropout turned on, you may see a higher training error.

For those familiar with the Lasagne framework built on top of Theano, there is an option something like: "get_output(net, deterministic = True)" (something like this, I forget exactly) where it does a deterministic forward pass, turning off Dropout and not performing any sort of noise injection.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.