0
$\begingroup$

I have coordinates data like: x_train = [7.6291 112.74 43.232 96.636 61.033 87.311 91.55 115.28 121.22 136.48 119.52 80.53 172.08 77.987 199.21 94.94 228.03 110.2 117.83 104.26 174.62 103.42 211.92 109.35 204.29 122.91 114.44 125.46 168.69 124.61 194.97 134.78 173.77 141.56 104.26 144.11 125.46 166.99 143.26 185.64 165.3 205.14]

Dimension is (1,42), total train samples are over (300000,42). I want to use Variational Autoencoder to train this dataset. In here, I am not sure KL divergence loss is appropriate for like this data. Because, when I train without KL loss (only mse loss), model is fine. But when I train with (KL + mse loss), loss did not decrease and not changing. That is why I want to know KL loss has a limitation on data or something?

$\endgroup$

closed as unclear what you're asking by Xi'an, mdewey, Peter Flom Mar 20 at 11:37

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

0
$\begingroup$

KL loss in VAE is similar to regularization strength - in Bayesian terms it could be weighted with how strong is your belief that the prior distribution is actually what you chose (normal distribution in VAE).

With that in mind, if your model fares worse using KL loss, you might just decrease its contribution to total loss:

Instead of $$\mathcal{L} = MSE + KL$$ use $$\mathcal{L'} = MSE + \beta KL$$

For some $\beta$ < 1. Some people called this approach $\beta$-VAE, for example you can check out this blogpost.

$\endgroup$
  • $\begingroup$ So, if I decrease KL contribution, should I give beta value < 1. Is there a proper way to choose beta value? By the way, when I give beta = 0.3, and train again. My reconstruction is still bad. $\endgroup$ – Dennis Thor Mar 20 at 10:25
  • $\begingroup$ It's actually the other way round - if you decrease $\beta$ you decrease KL-divergence contribution. You should most likely choose $\beta$ using crossvalidation, as you would do with any other hyperparameter $\endgroup$ – Jakub Bartczuk Mar 20 at 10:27
  • $\begingroup$ I am new to machine learning. So, could you provide me some links for crossvalidation for proper seed? $\endgroup$ – Dennis Thor Mar 20 at 10:39
  • $\begingroup$ Why do you use VAE then? Variational Bayes methods are very advanced, it's better to stick to simple stuff if you don't know much ML. For basics I'd recommend Andrew Ng's Coursera course. $\endgroup$ – Jakub Bartczuk Mar 20 at 21:45
  • $\begingroup$ I know about VAE and crossvalidation theory. I mean some sample code link for crossvalidation for NN. By the way, how do you think VAE is convenient on noisy data? $\endgroup$ – Dennis Thor Mar 21 at 5:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.