Questions tagged [generative-models]
A probabilistic or statistical model thought about as describing how the values in a sample is actually generated, and not only as a description or approximation.
302 questions
0
votes
0
answers
16
views
Why do VAEs work?
I am currently reading into Variational Autoencoders, and although I kind of understand the mathematical background described in the original paper (Auto-encoding Variational Bayes), I am struggling ...
2
votes
1
answer
66
views
Variational autoencoders - handcrafted example
In learning about variational autoencoders (VAEs), I would like to come up with a nice little handcrafted example to help make sense of them thoroughly. For this, suppose I know that my samples are ...
1
vote
0
answers
65
views
Is it possible to explain regression or classification, interpolation and generation using a single model structure?
Neural network is established as an universal approximator of all machine learning models. Further, double descent phenomenon in a neural network propagates the journey of regression to interpolation ...
- 81
0
votes
0
answers
59
views
How does MPGVAE calculate its reconstruction loss?
I recently came across the Graph Deconvolutional Generation paper where the author propose MPGVAE as a generative model for graphs. They present it as sort of an improvement of GraphVAE, with one ...
- 205
0
votes
0
answers
35
views
How to achieve Voice Conversion Using Voice Samples of a Specific Person using any voice as input?
I'm working on a project involving voice conversion, aiming to transform a voice to sound like a specific person speaking Darija (a Moroccan Arabic dialect). I have collected a set of voice samples ...
- 11
1
vote
1
answer
31
views
Background on notation for generative models
In many papers on generative modeling and Bayesian inference in statistics, I come across the following kind of notation, in particular for hierarchical models.
For variables $x_1, \dots, x_n$, it ...
- 11
1
vote
1
answer
46
views
Diffusion Model consistency term derivation question
The consistency term of the diffusion model is written as:
$$\mathop{\mathbb{E_{q_\phi(x_{1:T}|x_0)}}} \left[\log\prod_{t=2}^T \frac{p(x_{t-1} | x_t)}{q_\phi(x_{t-1}|x_t, x_0)}\right]$$
$$= \sum_{t=2}^...
1
vote
1
answer
69
views
How to derive Diffusion Model's reverse conditional probability when it's tractable via conditioning on $x_0$
Can anyone help me with understanding how the $\tilde{\beta}$ and ${\tilde\mu_t{(x_t, x_0)}}$ are derived?
It seems to me that exponential term is a 2nd order polynomial term and it doesn't really ...
- 11
3
votes
2
answers
383
views
In diffusion models (DDPM), if we predict the total noise, why not just remove the noise in one shot for sampling?
As pointed out by the DDPM paper, we can choose to reparameterize the prediction of the mean to prediction of the total noise "εθ is a function approximator intended to predict ε from x" (...
- 283
2
votes
1
answer
66
views
Why do we say that we're "predicting" the mean/noise in diffusion models?
In DDPM, ${\tilde\mu}_t$ is the mean of the conditional distribution $q(x_{t-1}|x_t,x_0)$ while the neural network $\mu_\theta$ is modeling a different conditional distribution $p_\theta(x_{t-1}|x_t)$....
- 283
2
votes
1
answer
114
views
Why sampling from the posterior is a good estimate for the Likelihood but sampling from the prior is bad?
In Variational Autoencoders (VAE), we have:
$$
\log p_\theta(x) = \log \left[ \int p_\theta(x \mid z)p(z) \, dz \right]
$$
where $ p_\theta(x \mid z) = \mathcal{N}(x; \mu_\theta(z), I) $ and $ p(z) = \...
- 328
5
votes
1
answer
296
views
Why do we care if the likelihood function is tractable?
I'm learning (deep) generative models and I've seen many places where the difficulty is that the likelihood function (with some parameters defined by our model) is intractable, i.e., it involves ...
- 283
1
vote
1
answer
57
views
NeRF vs mesh for text-to-3d generation
There seem to be multiple aproaches to generating 3d objects from text prompt. What's confusing is that some of them are generating NeRFs (https://arxiv.org/pdf/2308.16512), other's are generating ...
- 196
2
votes
2
answers
100
views
Why does Variational Inference work?
ELBO is a lower bound, and only matches the true likelihood when the q-distribution/encoder we choose equals to the true posterior distribution. Are there any guarantees that maximizing ELBO indeed ...
- 283
0
votes
0
answers
31
views
How are the MLE/MAP distinction and the generative/discriminative distinction related?
What is the relationship between Maximum Likelihood Estimation versus Maximum A Posteriori Estimation and generative modeling versus discriminative modeling? Is MLE an example of a generative model ...
- 133
1
vote
0
answers
29
views
How can I implement moment matching with kernel tricks if I do not have the complete distribution but only the higher-order moments or culuments?
As is said in Appendix B.3 of ref, "It is difficult to match high-order moments, because we have to deal with high order tensors directly. On the other hand, MMD can easily match high-order ...
0
votes
0
answers
16
views
Synthesizing multivariate time-series data with generative models for multiple data
I am new to multivariate time-series data and am particularly interested in synthesizing such data using generative models like TimeGAN: TimeGAN. From reading the literature on this topic, I have ...
- 101
0
votes
1
answer
54
views
Reverse distribution in DDPMs
In Denoising Diffusion Probabilistic Models, we want to reverse a forward process where we blend gaussian noise into a data point $\mathbf{x}_0$ over $T$ steps. The result of each forward step applied ...
- 101
0
votes
1
answer
28
views
How to do prediction (evaluate marginal likelihood) in generative latent variable classifier?
The dataset is $\{\boldsymbol x_t,y_t\}$ for $t=1,\dots,T$, where $y_t \in \{0,1\}$.
Define a generative latent variable classifier whose plate diagram is shown above.
For each data point, a local ...
- 201
0
votes
1
answer
67
views
About the use of Bayes' rule for continuous valued random variables
I am currently studying the book "An introduction to statistical learning with application in Python" and I am currently at the part 4 of chapter 4 where they explain the general framework ...
- 3
4
votes
2
answers
308
views
Why don’t diffusion models suffer posterior collapse?
In VAEs, posterior collapse occurs when the approximated posterior $q_\theta(z|x)$ becomes the standard Gaussian prior $p(z)$ after training (Lucas et al. 2019). The forward process of diffusion ...
- 41
0
votes
0
answers
10
views
Creating a light image generation model for a specific distribution
I am currently working on how a user can introduce bias in a neural network model.
To do so, I am creating an image2image model that only works on the training distribution.
For example, let's say I ...
- 19
0
votes
0
answers
138
views
VQ-VAE - Commitment Loss
Why the commitment loss is necessary in the VQ-VAE objective function?
I understand that it's serving the role of keeping the encoder outputs(continuous latent representations) close to the codebook.
...
- 1
0
votes
0
answers
32
views
My WGAN-GP isn't capturing bi-modal distributed underlying data
I am training a WGAN to generate a vector of 6 data points (6 dimensional output). I plotted a correlation matrix of my dataset to see the underlying distribution of the 6 data points, and the ...
- 1
2
votes
0
answers
173
views
Maximum Mean Discrepancy (MMD) implementation as a metric to measure GAN performance [closed]
I am trying to evaluate the performance of the GAN model, I have trained. I found that there exist two major choices FID (Fréchet inception distance) and MMD (Maximum Mean Discrepancy) for comparing ...
- 21
1
vote
0
answers
18
views
Help finding Bayesian model with multi-modal posterior [closed]
Background:
There is a paper (link) that concerns combining MCMC methods with a normalizing flow (a type of generative model). The basic idea is that the normalizing flow helps propose samples, which ...
- 31
0
votes
0
answers
68
views
Confused on Kullback-Leibler divergence being invoked without proper definition
I am trying to understand how authors of the DDPM paper in appendix A, made the leap from equation 21 to equation 22.
Specifically, it is not clear to me how they managed to convert the first term of ...
- 1,805
1
vote
0
answers
24
views
Can inverse sampling method be adapted to random vectors?
This might be a very basic question, but it seems that in all the examples I've seen, the inverse sampling method (i.e., input uniform RV into the inverse of CDF of desired PDF/probability ...
- 1,630
1
vote
0
answers
31
views
How does the order of implementation of Generative Data Augmentation and Generative Audio Super-Resoln matter? Which one should be implemented first?
I have a low quality audio dataset that I will use for classification. My goal is both to increase the quality of this dataset to make it easier to label it with supervised methods (super resolution) ...
- 23
1
vote
0
answers
25
views
Encoder-decoder Transformer model makes outputs predictions almost perfectly but fails to autoregressively decode
The model's sample predictions that I'm printing during training are almost perfect but the model generates meaningless tokens during evaluation.
For training I'm feeding it the source and target ...
- 4,177
0
votes
0
answers
862
views
How to calculate FID for a set with a small number of images?
I need to evaluate my generative model using FID (Fréchet inception distance). However, the dataset of real images that I have only contains 2719 examples. I've read that the authors of the metric ...
- 101
0
votes
1
answer
185
views
Why does higher dimensional data has higher likelihood?
I am reading about generative models. I came across an example a few times but I cannot come up with an explanation for it.
Imagine data is generated according to $p_\text{data}(x)$. It is often said ...
- 77
1
vote
0
answers
49
views
Why are LLMs generative models [duplicate]
According to Wikipedia:
A generative model is a statistical model of the joint probability distribution $P ( X , Y )$ on given observable variable $X$ and target variable $Y$;
A discriminative model ...
- 413
1
vote
1
answer
44
views
In what sense do Bayesian models simulate observation?
Could you verify that i understand the author correctly here? New to Bayes.
[...] These assumptions [the prior and the likelihood] also allow us to simulate the observations that the model implies. ...
- 111
4
votes
2
answers
351
views
Will synthetic data (e.g. from a generative model) produce valid statistical inference?
Especially with chatGPT now, there is a lot of interest in generative models for healthcare for creating synthetic patient data to amplify patient counts, e.g. for rare diseases. These are then ...
- 557
2
votes
1
answer
391
views
How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output
How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output. Also if my current understanding is right than we get same image for same value of z ...
- 21
1
vote
1
answer
456
views
What kind of Neural Networks are required for Diffusion models?
It appears that regular feed-forward and convolutions are not enough to make diffusion models work (from some personal limited testing, they do not work at all). The typical infrastructure was a U-Net ...
- 768
0
votes
1
answer
89
views
In factor VAE, do you freeze the discriminator weights during the back propagation step for the FVAE loss?
In factor vae, Disentangling by Factorising, there are two losses that are minimized. One is the VAE loss (eq. 2 in the paper) that includes (1) reconstruction loss, (2) KL divergence and (3) Total ...
- 33
0
votes
0
answers
26
views
What kind of learning is feature generation for score maximization?
What kind of deep learning is the generation of numerical features (Y) from objects (X) used to compute a score (f(.), differentiable) that is to be maximized directly?
Basically NN$\theta$(x) = y, f(...
2
votes
1
answer
40
views
Accuracy of probability estimate from generative autoregressive language model
My understanding is that a discriminative classifier such as a CNN that takes an input $x$ and produces a discrete output label $y$ is typically trained to predict the best value of $y$, and would not ...
4
votes
2
answers
122
views
What are the best resources on image synthesis?
What are some good resources to learn about image synthesis? What are some of the key concepts or architectures to study?
I understand image synthesis as generating new images with ML techniques.
- 485
4
votes
2
answers
293
views
Using Generative Adversarial Networks for joint distribution estimation
I am trying to use GAN model to generate N-dimensional samples with joint probability distribution that looks like some training data. I am having trouble getting the probability distribution of the ...
- 55
1
vote
0
answers
34
views
Expectation of energy-based model
An energy-based model parametrized by $\theta$ is defined as
$$
p(x; \theta) \propto \exp(-f(x; \theta))
$$
For my specific case, it is that $f(x; A, y) = - \langle x, Ax \rangle - \langle x, y \...
3
votes
1
answer
109
views
Blurring of image in generative model using diffusion probabilistic method
In the 2015 paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" by Sohl-Dickstein et al. on diffusion for generative models, Figure 1 shows the forward trajectory for a 2-...
0
votes
1
answer
114
views
VAE active units
According to Burda et al (2015) number of active units is computed as:
$ Cov_x(E_{z \sim q_\phi(z|x)}) > \delta $
for some particular delta. In the paper it is set to 0.02 empirically. But this ...
1
vote
1
answer
33
views
Rewrite linear expression
Is it possible to rewrite
$$
\frac{-1}{2}\left(x^T\Gamma^{-1}(\mu_1-\mu_0)+(\mu_1-\mu_0)^T\Gamma^{-1}x\right)
$$
as
$$
-\theta^Tx
$$
where $x,\mu\in\mathbb{R}^d, \Gamma\in\mathbb{R}^{d\times d}$ and $\...
- 13
0
votes
1
answer
102
views
Normalizing Flows Invertibility
I am currently reading up on RealNVP, which has the following transformations according Lilian Weng:
$$
\begin{aligned}
\mathbf{y}_{1:d} &= \mathbf{x}_{1:d} \\
\mathbf{y}_{d+1:D} &= \mathbf{x}...
- 19
10
votes
6
answers
2k
views
Why do Denoising Diffusion Probabilistic Models (DDPM) add noise according to $\sigma_t$ during sampling?
Reading about Denoising Diffusion Probabilistic Models (DDPM) the paper (algorithm 2 - sampling) states that the sampling goes according to...
$$
x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{...
- 962
2
votes
1
answer
1k
views
How to generate new data with a VAE?
I have built the following function which takes as input some data and runs a VAE on them:
...
- 531
4
votes
1
answer
1k
views
How to rewrite DreamBooth loss in terms of $\epsilon$-prediction?
I'm trying to make the loss used in DreamBooth paper explicit, writing it in terms of the noise, as it is commonly written in the original diffusion article [1], instead of the image reconstruction ...
- 505