0
$\begingroup$

I'm trying to make a variational autoencoder with PyTorch that generates made-up pronounceable words. I'm following some tutorials and using this code as an example.

The training data is the 10,000 most common English words. The neural network's input is a flattened one-hot encoding of the training data and the length of the word. The words' lengths range from 3 characters to 10 characters. If the word is shorter than 10 characters, zero-padding is used. Example:

apple

[5, # length of word, used for generating

 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,
 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,
 0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
 0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,

 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
 ...
 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
]

The issue is that there's barely any variation in the generated words. When random numbers are given to the decoder, it always seems to output these same lame words:

["seaeee",
 "seaeee",
 "seaeer",
 "sereee",
 "seaeee",
 ...

I experimented with various sizes of the hidden layers and the latent vector, but it made no difference.

I tried making every layer have the same number of neurons as the input, so the data should pass straight through the network without a bottleneck, but that still didn't work.

I tried adjusting the weights of BCE and KLD in the loss function as described here, but even that had no effect.

What could be causing my VAE to generate the same outputs all the time?


Update: New Results

['soaeer', 'xjzzjq', 'sereee', 'xjzqqjw', 'seoeee',
 'seaeee', 'xjzqjq', 'aoeeee', 'dfetsr', 'seaeee',
 'soaeer', 'sateer', 'seaeer', 'xjzqjjw', 'soeeee',
 'seaeee', 'xjzqqjw', 'seaeee', 'soaeee', 'seaeer',
 'xjqzqjw', 'seaeee', 'soaeer', 'rietrn', 'xjzqjjw', 
 'seaeee', 'sooeer', 'seoeee', 'seeeee', 'seaeee', 
 'sereee', 'seaeee', 'seoeee', 'seoeee', 'soeeee',
 'oolnry', 'magdueo', 'seeeee', 'seoeee', 'lestee',
 'darnre', 'soaeee', 'seaeee', 'sooeer', 'seaeee',
 'seteee', 'taoere', 'soaeee', 'seaeee', 'xjzqqj',
 'seaeee', 'seaeee', 'seaeee', 'seaeee', 'seoeee',
 'aoeeer', 'bplswn', 'seeeee', 'seaeee', 'seoeee',
 'seaeee', 'soaeee', 'xjzqqjw', 'sooeee', 'seaeen',
 'xjbzvv', 'seeeee', 'soreee', 'soaeee', 'soaeee',
 'rwlarn', 'seaeee', 'xjqzjp', 'seaeee', 'xjzqqjw',
 'seoeee', 'sareet', 'seeeee', 'seaeee', 'seaera',
 'seaeee', 'seaeee', 'seaeee', 'saoeer', 'seaeee',
 'seaeee', 'seaeee', 'seaeee', 'seaeee', 'xjqzjpj',
 'seeeee', 'soreee', 'seaeee', 'seoeee', 'seaeee',
 'seoeee', 'saneee', 'vyceor', 'saaeer', 'caaeee']
$\endgroup$