3
Bridgeburners's comment is correct: in this model, softmax always predicts 1 (because there's only one output node). I changed to sigmoid activation as I had used originally, then found another problem: under sigmoid, model.predict returns probabilities, not classes, so the predictions need to thresholded first. With these bugs fixed, and epochs cranked up ...
2
It must be a function of its inputs. It can be univariate or multivariate (for example, softmax). The more useful ones are often non-constant and continuous.
(Approximately) Monotonic functions have been found, empirically, to be better.
A few very successful activations are not monotonic however (Swish for example).
It does not even need to be one to one ...
1
That looks like a bug. I tweaked the same repo, and used the standard toy "8-gaussians" dataset, and trained a VAE.
Red points: samples from the 8-gaussians dataset
Blue points: sampled from the trained model
It's nowhere near perfect (and 8-gaussians is actually a non-trivial task for many other generative models), but the mass is concentrated in the ...
1
Batch norm as the last layer of the encoder isn't technically wrong, but it is likely to be a bad idea (in general, never use batch norm as the last layer). And you can see in the github link referenced, that the results from that model were pretty poor due to this.
Relu for std/variance could be valid, if you decided to directly predict std/variance ...
1
So here is my answer. This is by no means the best answer. Rather it is just a basic general framework that is used.
When I fit a model I typically like to decide the "best fit" by splitting my data into a training set and a validation set. The training set is used to actually fit the model and the validation set is used to test the model's accuracy. The ...
1
You apply dropout after the non-linear activation function.
Sources for this:
https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf - Formula on
page 1933 and diagram on the next page.
https://sebastianraschka.com/faq/docs/dropout-activation.html
https://pgaleone.eu/deep-learning/regularization/2017/01/10/anaysis-of-dropout/
Only top voted, non community-wiki answers of a minimum length are eligible
