Questions tagged [autoencoders]
Feedforward neural networks trained to reconstruct their own input. Usually one of the hidden layers is a "bottleneck", leading to encoder->decoder interpretation.
639
questions
0
votes
0
answers
9
views
What is the expectation of P(x|z) under distribution of z parameterized by x?
This question stems from Section 2.1 of this VAE tutorial. The problem stated in the paper is to compute the data likelihood using law of total probability:
$$
P(X) = \int P(X,z) \,\mathrm dz = \int P(...
- 41
0
votes
0
answers
10
views
VAEs for omics data [closed]
I am training a VAE network with Cyclic Immunofluorescence (CyCIF) images of cancer tissues. These are huge images (can be more than 10000px by 10000px) and have several channels. In our case, 38 ...
- 101
-1
votes
0
answers
9
views
What type of Autoencoder is this?
I have an autoencoder structure which is given below. Now, I want to know what type of autoencoder is this? Is it Bayesian or Gaussian?? Or is it something else?
...
- 1
0
votes
0
answers
5
views
increasing the capacity of an autoencoder
I have an autoencoder model with 5 layers in encoding and 5 layers in the decoding section. I am using this model for signal processing the problem is that it is making the signal way more smooth that ...
- 1
0
votes
0
answers
19
views
Trained network always predicts zero [duplicate]
I have an encoder model and I'm training it with a dataset of signals with size (500,1). The data set is normalized and then used to train the model but the problem is that after the model is trained, ...
- 1
1
vote
1
answer
36
views
0
votes
0
answers
18
views
VAE for Motion Sequence Generation - Convergence Issue with Scheduled Sampling
I implemented a Variational Autoencoder (VAE) in PyTorch for motion sequence generation using human pose data (joint angles and angular velocities in radians) from the CMU dataset. The VAE ...
- 1
0
votes
0
answers
20
views
On the expressivity of latent variable models
Empirically, we have seen that VAEs can approximate very complex distributions. I am interested in knowing if there are any theoretical results showing how expressive latent variable models can be. ...
0
votes
0
answers
38
views
KL Divergence in VAE [duplicate]
The basic KL-Divergence between two distributions is as:
$KL(N(\mu_1,\sigma_1) || N(\mu_2, \sigma_2)) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{...
- 225
0
votes
1
answer
63
views
Image pre-processing for Variational Autoencoder
Setting
I am training a Variational Autoencoder (VAE) on the CIFAR10 dataset, which has RGB colors. The VAE uses convolution and transposed convolution layers as well as linear layers to encoder and ...
- 33
0
votes
0
answers
12
views
Deconvolution vs tf.Reshape
I currently have a 1D-CNN which produces a 1D output due to the dense layers at the end of said CNN but want it to produce a 2D output. Instead of reshaping my tensor elements using tf.reshape, would ...
- 1
0
votes
1
answer
41
views
Understanding a beta-variational autoencoder
I'm working on a beta-variational autoencoder using car images from the Vehicle Color Recognition Dataset. At this point, I'm just exploring different architectures and values for beta. (If you're ...
- 1,128
1
vote
0
answers
32
views
Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?
I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though $q_{\phi}(z|x)$ and $p_{\theta}(x|z)$ are Gaussian.
I understand that the parameters ...
- 21
0
votes
0
answers
11
views
How can a linear autoencoder with $h=1$ hidden unit reconstruct any rank 1 matrix?
I've had this as a homework problem as a true or false type of question and I'm trying to wrap my head around why this is true.
Is the reason simply represent each datapoint as a scaled version of a ...
- 290
0
votes
0
answers
27
views
How do I check that two ways of expressing the same random variable lead to the same distribution?
When reading the explanation for the "The reparametrization trick" on the Stanford's cs228 notes, I saw a claim that
It is easy to check that the two ways of expressing the random
variable ...
- 129
0
votes
0
answers
21
views
Can a linear autoencoder perform projection as PCA with less layers than components? [duplicate]
Assume that you have a matrix $X$ and you want to project it onto a subspace by using PCA. It will work. Then you are trying to use a linear autoencoder to projecting $X$ onto the same subspace. It ...
- 381
1
vote
0
answers
14
views
A ML classifier for predicting the hourly direction for a group of stocks where training stocks don't match out of sample stocks [closed]
I have historical data for 100 stocks (call them A). I would like to train a model jointly on all stocks which will learn from the cross sectional historical activity, to predict another set of stocks ...
- 121
0
votes
0
answers
11
views
Linear autoencoders - Will they only preserve linear separable data?
I'm looking for to compress an image $X$ into a smaller image $x$. But not only compress, also reduce its view into a simplier view that are linear separable.
My question is: Can I use linear ...
- 381
0
votes
0
answers
28
views
Implementing a collaborative filtering model for recommending items to new users and how to evaluate it
I have some data for a collaborative filtering problem, which can be represented in a user-item interaction matrix. The data consists of 20 million orders, where there are 15000 different items, and ...
1
vote
1
answer
37
views
How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output
How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output. Also if my current understanding is right than we get same image for same value of z ...
- 11
0
votes
0
answers
23
views
Prior estimation in Dynamic (sequence to sequence) Variational Autoencoders (DVAE) with LSTMs
I am trying to implement a sequence-to-sequence variational autoencoder that consists of two parallel sequence encoders. One of the encoders is based on a standard normal prior as in the vanilla vae (...
- 81
0
votes
2
answers
284
views
How to Resolve Variational Autoencoder (VAE) Model Collapse in Reconstruction Task Using Sensor Data?
I am currently experiencing a suspected model collapse in a Variational Autoencoder (VAE) model I am working with. Below are details on the project setup and the issue at hand:
Project Goal: Exploring ...
2
votes
1
answer
32
views
What purpose do higher dimensional mappings serve compared to lower dimensional mappings?
In Neural Network Architectures I understand that lower dimensional mappings(for example mapping and input to a space of lower dimension) can serve the purpose of decreasing dimensionality while ...
0
votes
2
answers
43
views
Rank Neurons Importance of the latent space of an Autoencoder using PCA
I am trying to extract only the important neurons from the latent space of an Autoencoder to be converted later to a pattern for a model pattern recognizer.
PCA Loadings helps in finding the highest ...
0
votes
1
answer
54
views
In factor VAE, do you freeze the discriminator weights during the back propagation step for the FVAE loss?
In factor vae, Disentangling by Factorising, there are two losses that are minimized. One is the VAE loss (eq. 2 in the paper) that includes (1) reconstruction loss, (2) KL divergence and (3) Total ...
- 23
1
vote
0
answers
49
views
Is it correct to do SVD from the latent space of an autoencoder?
Is it correct to do SVD from the latent space of an autoencoder?
I am asking because I think that by performing SVD from a latent space, and plotting the singular values, it is possible to know the ...
- 11
1
vote
0
answers
15
views
VAE: Noisy decoder output
I'm trying to implement a simple VAE by following several tutorials
like this and this. This is the code that I came up with:
...
- 11
0
votes
0
answers
31
views
VAE with only forward diffusion enhancement ** experiment **
I wanted to get some opinions with an idea that I have explored for a little bit. This is an experiment and I would like to know if this is mathematically plausible or not.
Imagine $\bar{x}$ is the ...
- 643
1
vote
0
answers
27
views
How does Variational Autoencoder approximate the joint probability distribution?
I know that in Variational Inference the idea is to approximate the posterior P(z|x, y) and I know that Variational AutoEncoders (VAEs) use the idea of variational inference through neural network ...
- 143
0
votes
0
answers
115
views
Should the KL loss term for a VAE be the KL-Loss of a batch's mean mu and log sigma, or is it the mean of the kl loss for each individual input image?
I've been trying to learn about Variational Autoencoders and been looking at the Keras sample implementation (https://github.com/keras-team/keras-io/blob/master/examples/generative/vae.py)
I'm ...
- 121
0
votes
1
answer
65
views
Why does the sign of the loss in VAEs appear to be backwards?
I'm trying to fully understand Variational Autoencoders (VAEs) 1 and their math but one part keeps confusing me and I hope someone can give me in an intuitive explanation what I am missing.
Here is ...
- 1
1
vote
0
answers
30
views
Weighting feature-specific reconstruction loss in Seq2Seq VAE
I am working on training a Seq2Seq Variational Autoencoder (VAE) model using healthcare data. In my dataset, I have features that exhibit varying levels of variance across patients. For instance, ...
2
votes
1
answer
105
views
What is a Student-t VAE ? and how is it different from Gaussian VAE?
I am currently reading https://www.ijcai.org/proceedings/2018/0374.pdf ,this is a research paper based on Student-t Variational Autoencoder for Robust Density Estimation , In this research paper, they ...
- 21
0
votes
0
answers
19
views
Constant terms in variational inference and lower bounds
I have a model for which the lower bound takes the following form
$
\log p(x) \geq \mathbb{E}_{recog}[\log \frac{gen}{recog}] + \mathcal{c}
$
where recog and gen denote the factorization in the ...
- 113
3
votes
1
answer
502
views
PCA via a Neural Network
In a simple neural network, having more nodes on an input layer that on the next layer performs a compression or dimension reduction similar to what PCA does. The fewer nodes encode in a combination ...
- 1,863
0
votes
1
answer
58
views
VAE active units
According to Burda et al (2015) number of active units is computed as:
$ Cov_x(E_{z \sim q_\phi(z|x)}) > \delta $
for some particular delta. In the paper it is set to 0.02 empirically. But this ...
4
votes
1
answer
684
views
Source of randomness in ChatGPT
I've read that ChatGPT will sometimes give different answers to the same prompt. In other words, there is an element of randomness. Where does this randomness come from? Is there some sort of ...
- 6,297
3
votes
2
answers
194
views
Replacing the KL-divergence term in a VAE with parameter regularization
When training a VAE, one aim to optimize function $\mathcal{L}$, defined as:
$$\mathcal{L}\left(\theta,\phi; \mathbf{x}^{(i)}\right) = - D_{KL}\left(q_\phi(\mathbf{z}|\mathbf{x}^{(i)}) || p_\theta(\...
- 936
1
vote
1
answer
88
views
Calculating KL divergence with entropy and cross entropy for VAEs
When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is:
$$ KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2) $$
or some variation ...
- 23
1
vote
0
answers
63
views
How to measure posterior collapse if any
Is there any theoretical work on how to measure posterior collapse?
One can measure decoder output, but it is not clear if the degradation (if any) happened due to posterior collapse or due to failing ...
1
vote
0
answers
112
views
Is it acceptable to use pooling layers in variational autoencoders?
When training a model for image classification it is common to use pooling layers to reduce the dimensionality, as we only care about the final node values corresponding to the categorical ...
- 23
1
vote
0
answers
86
views
In the β-TCVAE paper, can someone help with the derivation (S3) in Appendix C.1?
Paper: Isolating Sources of Disentanglement in VAEs
I follow as far as,
$$\mathbb{E}_{q(z)}[log[q(z)] = \mathbb{E}_{q(z, n)}[\ log\ \mathbb{E}_{n'\sim\ p(n)}[q(z|n')]\ ]$$ Subsequently, I don't ...
- 23
0
votes
0
answers
62
views
Training Autoencoder on time series with repeating pattern
Im training an autoencoder on a time series that consists of repeating patterns (because the same process is repeated again and again). If I then use this autoencoder to reconstruct another one of ...
- 21
0
votes
0
answers
38
views
Graph based variational Autoencoder with variable latent size
I'm trying to build a graph-based Variational-Autoencoder, which should be able to generate graph structures (adjacency matrices). So far, all the papers and models I've seen use a fixed latent vector ...
- 21
0
votes
0
answers
309
views
Variational Autoencoder - What's the best way to output and which loss function? [duplicate]
I'm hoping this is the right place to ask. I'm new to deep learning and especially variational NNs.
I'm trying to create a VAE using tensorflow/tensorflow_probability which can recreate a gaussian ...
- 1
0
votes
0
answers
26
views
Larger Latent Space Dim for Point Cloud Autoencoder
So I'm trying to follow a paper that uses a AE to learn point clouds. The thing is, the dimension of the point cloud data is 3 (x, y, z), but the dimension of the latent space from what I can tell is ...
- 101
0
votes
0
answers
52
views
Parametrize the variance of the gaussian posterior distribution in vae
I noticed that in most of the implementations of a variational autoencoder with gaussian posterior, the variance of the gaussian is not learned during training. The decoder usually outputs only the ...
- 1
0
votes
1
answer
36
views
Activation function selection for autoencoders
I am running an autoencoder model and have to select my activation function. I have structured data (i.e. not an image) which comprises continuous variables which have a range of -4 to +4 for each ...
- 117
2
votes
0
answers
158
views
Why is the Wasserstein distance not used in Variational Inference
I just started learning the concept of variational inference in the context of variational Autoencoder, so please excuse me if the answer is obvious. I would like to know why traditionally, KL-...
- 21
1
vote
0
answers
246
views
How exactly does the weight applied to the KL divergence in $\beta$-VAE lead to disentanglement?
In "β-VAE: LEARNING BASIC VISUAL CONCEPTS WITH A CONSTRAINED VARIATIONAL FRAMEWORK", the Introduction states that "With β > 1 the model is pushed to learn a more efficient latent ...
- 23