Timeline for Is it possible to make multi-layer autoencoder learn to completely repeat input?
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 25, 2017 at 10:05 | vote | accept | dk14 | ||
| Mar 25, 2017 at 10:04 | comment | added | dk14 | actually, maybe there is some problem with my new configuration/code. I tried to use only one layer and got same results. I think I should re-check it again... | |
| Mar 25, 2017 at 9:52 | comment | added | dk14 |
after applying LeakyRELU I've got rid of zeros but they were replaced with small negative values [0.6623673,0.12019485,0.13492113,0.57845426] ==> [-5.88048E-4,0.12378041,-0.00475002,-0.0019856603]
|
|
| Mar 25, 2017 at 9:39 | history | edited | Hugh Perkins | CC BY-SA 3.0 |
added 587 characters in body
|
| Mar 25, 2017 at 9:37 | comment | added | Hugh Perkins | (I guess leaky relu might do what you want: piecewise linear, so gradients wont vanish horribly, has gradient almost everywhere, is technically an activation function (cf just not having an activation function at all...), and is quite commonly used) | |
| Mar 25, 2017 at 9:34 | comment | added | Hugh Perkins | Hmmm. Interesting that works for one layer but not for multiple. Maybe whats happening is that ReLU is blocking your gradient backprop. I'd be tempted to try 1. using an activaiton function that has non-zero gradient almost everywhere, eg ELU, or leaky ReLU; and/or 2. adding dropout (to cause the neurons to be turned on/off randomly, albeit the noise will mean the network takes longer to converge). | |
| Mar 25, 2017 at 9:32 | comment | added | dk14 | Thanks for the reply. 1) my values are strictly positive to avoid ReLU "saturation" as you could see from example 2) basically my problem is that I can do it with a single layer easily (because of no back-propagation there) - I want to see same behaviour for back-propagated layers, not for stacked ones... or at least get some explanation why it doesn't work with backprop. | |
| Mar 25, 2017 at 9:03 | history | answered | Hugh Perkins | CC BY-SA 3.0 |