Deep Learning : Using dropout in Autoencoders?

Question

I am working with autoencoders and have few confusions, I am trying different autoencoders like:

• fully_connected autoencoder
• convolutional autoencoder
• denoising autoencoder

I have two datasets, one is numerical with float and int values, second is a text dataset with text and date values.

Numerical dataset looks like this:

date ,        id ,             check_in , check_out , coke_per , permanent_values , temp
13/9/2017     142453390001    134.2       43.1        13         87                 21
14/9/2017     142453390005    132.2       46.1        19         32                 41
15/9/2017     142453390002    120.2       42.1        33         99                 54
16/9/2017     142453390004    100.2       41.1        17         39  

Any my text dataset looks like this:

data              text
13/9/2017         i totally understand this conversation about farmer market and the organic products, a nice conversation ’cause prices are cheaper than traditional
14/9/2017         The conversation was really great. But I think I need much more practice. I need to improve my listening a lot. Now I’m very worried because I thought that I’d understand more. Although, I understood but I had to repeat and repeat. See you!!!

My questions are:

1. Should I normalize my numerical data values before feeding to any type of autoencoder? If they are int and float values do I still have to normalize?

2. Which activation function should I use in autoencoder? Some article and research paper says "sigmoid" and some says "relu"?

3. Should I use dropout in each layer? Like if my artichare for autoencoder looks like

   encoder (1000 --> 500 -- > 256 ----> 128 ) --> decoder (128 --> 256 --> 500--> 784)

something like this?

encoder(dropout(1000,500) --> dropout( 500,256) --> dropout (256,128) )----> decoder(dropout(128,256),dropout(256,500),dropout(500,784))

4. For text dataset, if I am using word2vec or any embedding to convert text into vector then I would have float values for each word, should I normalize that data too?

   text ( Hello How are you ) -- > word2vec(text) ----> ([1854.92002 , 54112.89774 ,5432.9923 ,5323.98393])

Should I normalize these values or directly use them in autoencoder ?

Answer 1

Should i normalize my numerical data values before feeding to any type of autoencoder? If they are int and float values do I still have to normalize?

Normalizing data often improves the model because it amounts to pre-conditioning the inputs so that optimization proceeds more smoothly.

Which activation function should I use in autoencoder? Some article and research paper says "sigmoid" and some says "relu"?

Use the one that works best for your problem. ReLUs are gaining popularity because they alleviate some problems with sigmoids units. See What are the advantages of ReLU over sigmoid function in deep neural networks? for some more information.

Should I use dropout in each layer?

That depends on what you want your model to do and what qualities you want it to have. Autoencoders that include dropout are often called "denoising autoencoders" because they use dropout to randomly corrupt the input, with the goal of producing a network that is more robust to noise. This tutorial has more information.