Skip to main content

Questions tagged [generative-models]

A probabilistic or statistical model thought about as describing how the values in a sample is actually generated, and not only as a description or approximation.

Filter by
Sorted by
Tagged with
0 votes
0 answers
3 views

In diffusion models (DDPM), if we predict the total noise, why not just remove the noise in one shot for sampling?

As pointed out by the DDPM paper, we can choose to reparameterize the prediction of the mean to prediction of the total noise "εθ is a function approximator intended to predict ε from x" (...
Daniel Mendoza's user avatar
0 votes
0 answers
19 views

Why do we say that we're "predicting" the mean/noise in diffusion models?

In DDPM, ${\tilde\mu}_t$ is the mean of the conditional distribution $q(x_{t-1}|x_t,x_0)$ while the neural network $\mu_\theta$ is modeling a different conditional distribution $p_\theta(x_{t-1}|x_t)$....
Daniel Mendoza's user avatar
2 votes
1 answer
88 views

Why sampling from the posterior is a good estimate for the Likelihood but sampling from the prior is bad?

In Variational Autoencoders (VAE), we have: $$ \log p_\theta(x) = \log \left[ \int p_\theta(x \mid z)p(z) \, dz \right] $$ where $ p_\theta(x \mid z) = \mathcal{N}(x; \mu_\theta(z), I) $ and $ p(z) = \...
rando's user avatar
  • 308
5 votes
1 answer
219 views

Why do we care if the likelihood function is tractable?

I'm learning (deep) generative models and I've seen many places where the difficulty is that the likelihood function (with some parameters defined by our model) is intractable, i.e., it involves ...
Daniel Mendoza's user avatar
1 vote
1 answer
14 views

NeRF vs mesh for text-to-3d generation

There seem to be multiple aproaches to generating 3d objects from text prompt. What's confusing is that some of them are generating NeRFs (https://arxiv.org/pdf/2308.16512), other's are generating ...
zlenyk's user avatar
  • 186
2 votes
2 answers
62 views

Why does Variational Inference work?

ELBO is a lower bound, and only matches the true likelihood when the q-distribution/encoder we choose equals to the true posterior distribution. Are there any guarantees that maximizing ELBO indeed ...
Daniel Mendoza's user avatar
0 votes
0 answers
15 views

How are the MLE/MAP distinction and the generative/discriminative distinction related?

What is the relationship between Maximum Likelihood Estimation versus Maximum A Posteriori Estimation and generative modeling versus discriminative modeling? Is MLE an example of a generative model ...
user10478's user avatar
  • 123
1 vote
0 answers
13 views

How can I implement moment matching with kernel tricks if I do not have the complete distribution but only the higher-order moments or culuments?

As is said in Appendix B.3 of ref, "It is difficult to match high-order moments, because we have to deal with high order tensors directly. On the other hand, MMD can easily match high-order ...
Kaiming Zhang's user avatar
0 votes
0 answers
12 views

Synthesizing multivariate time-series data with generative models for multiple data

I am new to multivariate time-series data and am particularly interested in synthesizing such data using generative models like TimeGAN: TimeGAN. From reading the literature on this topic, I have ...
user3178756's user avatar
0 votes
1 answer
20 views

Reverse distribution in DDPMs

In Denoising Diffusion Probabilistic Models, we want to reverse a forward process where we blend gaussian noise into a data point $\mathbf{x}_0$ over $T$ steps. The result of each forward step applied ...
CoconutFred's user avatar
0 votes
1 answer
23 views

How to do prediction (evaluate marginal likelihood) in generative latent variable classifier?

The dataset is $\{\boldsymbol x_t,y_t\}$ for $t=1,\dots,T$, where $y_t \in \{0,1\}$. Define a generative latent variable classifier whose plate diagram is shown above. For each data point, a local ...
Kaiwen's user avatar
  • 81
0 votes
1 answer
37 views

About the use of Bayes' rule for continuous valued random variables

I am currently studying the book "An introduction to statistical learning with application in Python" and I am currently at the part 4 of chapter 4 where they explain the general framework ...
Vincent's user avatar
0 votes
0 answers
91 views

Why don’t diffusion models suffer posterior collapse?

In VAEs, posterior collapse occurs when the approximated posterior $q_\theta(z|x)$ becomes the standard Gaussian prior $p(z)$ after training (Lucas et al. 2019). The forward process of diffusion ...
entropy07's user avatar
0 votes
0 answers
9 views

Creating a light image generation model for a specific distribution

I am currently working on how a user can introduce bias in a neural network model. To do so, I am creating an image2image model that only works on the training distribution. For example, let's say I ...
Adrien's user avatar
  • 19
0 votes
0 answers
87 views

VQ-VAE - Commitment Loss

Why the commitment loss is necessary in the VQ-VAE objective function? I understand that it's serving the role of keeping the encoder outputs(continuous latent representations) close to the codebook. ...
mcr0yal's user avatar
0 votes
0 answers
32 views

My WGAN-GP isn't capturing bi-modal distributed underlying data

I am training a WGAN to generate a vector of 6 data points (6 dimensional output). I plotted a correlation matrix of my dataset to see the underlying distribution of the 6 data points, and the ...
ShrHon's user avatar
  • 1
2 votes
0 answers
88 views

Maximum Mean Discrepancy (MMD) implementation as a metric to measure GAN performance [closed]

I am trying to evaluate the performance of the GAN model, I have trained. I found that there exist two major choices FID (Fréchet inception distance) and MMD (Maximum Mean Discrepancy) for comparing ...
Rajesh Nakka's user avatar
1 vote
0 answers
11 views

Help finding Bayesian model with multi-modal posterior [closed]

Background: There is a paper (link) that concerns combining MCMC methods with a normalizing flow (a type of generative model). The basic idea is that the normalizing flow helps propose samples, which ...
caitlin's user avatar
  • 31
0 votes
0 answers
62 views

Confused on Kullback-Leibler divergence being invoked without proper definition

I am trying to understand how authors of the DDPM paper in appendix A, made the leap from equation 21 to equation 22. Specifically, it is not clear to me how they managed to convert the first term of ...
Spacey's user avatar
  • 1,805
1 vote
0 answers
20 views

Can inverse sampling method be adapted to random vectors?

This might be a very basic question, but it seems that in all the examples I've seen, the inverse sampling method (i.e., input uniform RV into the inverse of CDF of desired PDF/probability ...
Fraïssé's user avatar
  • 1,590
1 vote
0 answers
29 views

How does the order of implementation of Generative Data Augmentation and Generative Audio Super-Resoln matter? Which one should be implemented first?

I have a low quality audio dataset that I will use for classification. My goal is both to increase the quality of this dataset to make it easier to label it with supervised methods (super resolution) ...
Yalçın Cenik's user avatar
2 votes
0 answers
21 views

Encoder-decoder Transformer model makes outputs predictions almost perfectly but fails to autoregressively decode

The model's sample predictions that I'm printing during training are almost perfect but the model generates meaningless tokens during evaluation. For training I'm feeding it the source and target ...
Sean's user avatar
  • 4,077
0 votes
0 answers
29 views

VAE for Motion Sequence Generation - Convergence Issue with Scheduled Sampling

I implemented a Variational Autoencoder (VAE) in PyTorch for motion sequence generation using human pose data (joint angles and angular velocities in radians) from the CMU dataset. The VAE ...
RTn's user avatar
  • 1
0 votes
0 answers
647 views

How to calculate FID for a set with a small number of images?

I need to evaluate my generative model using FID (Fréchet inception distance). However, the dataset of real images that I have only contains 2719 examples. I've read that the authors of the metric ...
nietoperz21's user avatar
0 votes
0 answers
20 views

Should I account for email opens when modeling link clicks embedden in the email?

I'm trying to build a model that predicts whether an email recipient will click on a link inside an email. As input, it takes member-related email click/open/unsubscribe history, as well as that ...
Jake's user avatar
  • 199
0 votes
1 answer
154 views

Why does higher dimensional data has higher likelihood?

I am reading about generative models. I came across an example a few times but I cannot come up with an explanation for it. Imagine data is generated according to $p_\text{data}(x)$. It is often said ...
Alf's user avatar
  • 77
1 vote
0 answers
47 views

Why are LLMs generative models [duplicate]

According to Wikipedia: A generative model is a statistical model of the joint probability distribution $P ( X , Y )$ on given observable variable $X$ and target variable $Y$; A discriminative model ...
Sam's user avatar
  • 393
1 vote
1 answer
37 views

In what sense do Bayesian models simulate observation?

Could you verify that i understand the author correctly here? New to Bayes. [...] These assumptions [the prior and the likelihood] also allow us to simulate the observations that the model implies. ...
rtviii's user avatar
  • 111
4 votes
2 answers
309 views

Will synthetic data (e.g. from a generative model) produce valid statistical inference?

Especially with chatGPT now, there is a lot of interest in generative models for healthcare for creating synthetic patient data to amplify patient counts, e.g. for rare diseases. These are then ...
elbord77's user avatar
  • 557
2 votes
1 answer
234 views

How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output

How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output. Also if my current understanding is right than we get same image for same value of z ...
Hamit Des's user avatar
1 vote
1 answer
316 views

What kind of Neural Networks are required for Diffusion models?

It appears that regular feed-forward and convolutions are not enough to make diffusion models work (from some personal limited testing, they do not work at all). The typical infrastructure was a U-Net ...
Edv Beq's user avatar
  • 713
0 votes
1 answer
79 views

In factor VAE, do you freeze the discriminator weights during the back propagation step for the FVAE loss?

In factor vae, Disentangling by Factorising, there are two losses that are minimized. One is the VAE loss (eq. 2 in the paper) that includes (1) reconstruction loss, (2) KL divergence and (3) Total ...
S R's user avatar
  • 33
0 votes
0 answers
25 views

What kind of learning is feature generation for score maximization?

What kind of deep learning is the generation of numerical features (Y) from objects (X) used to compute a score (f(.), differentiable) that is to be maximized directly? Basically NN$\theta$(x) = y, f(...
user389279's user avatar
2 votes
1 answer
39 views

Accuracy of probability estimate from generative autoregressive language model

My understanding is that a discriminative classifier such as a CNN that takes an input $x$ and produces a discrete output label $y$ is typically trained to predict the best value of $y$, and would not ...
sunfishstanford's user avatar
4 votes
2 answers
114 views

What are the best resources on image synthesis?

What are some good resources to learn about image synthesis? What are some of the key concepts or architectures to study? I understand image synthesis as generating new images with ML techniques.
Glue's user avatar
  • 485
4 votes
2 answers
245 views

Using Generative Adversarial Networks for joint distribution estimation

I am trying to use GAN model to generate N-dimensional samples with joint probability distribution that looks like some training data. I am having trouble getting the probability distribution of the ...
dvd8719's user avatar
  • 55
1 vote
0 answers
33 views

Expectation of energy-based model

An energy-based model parametrized by $\theta$ is defined as $$ p(x; \theta) \propto \exp(-f(x; \theta)) $$ For my specific case, it is that $f(x; A, y) = - \langle x, Ax \rangle - \langle x, y \...
Vũ Lê Thế Anh's user avatar
3 votes
1 answer
102 views

Blurring of image in generative model using diffusion probabilistic method

In the 2015 paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" by Sohl-Dickstein et al. on diffusion for generative models, Figure 1 shows the forward trajectory for a 2-...
sunfishstanford's user avatar
0 votes
1 answer
92 views

VAE active units

According to Burda et al (2015) number of active units is computed as: $ Cov_x(E_{z \sim q_\phi(z|x)}) > \delta $ for some particular delta. In the paper it is set to 0.02 empirically. But this ...
Pavel Podlipensky's user avatar
1 vote
1 answer
33 views

Rewrite linear expression

Is it possible to rewrite $$ \frac{-1}{2}\left(x^T\Gamma^{-1}(\mu_1-\mu_0)+(\mu_1-\mu_0)^T\Gamma^{-1}x\right) $$ as $$ -\theta^Tx $$ where $x,\mu\in\mathbb{R}^d, \Gamma\in\mathbb{R}^{d\times d}$ and $\...
jroc's user avatar
  • 13
0 votes
1 answer
90 views

Normalizing Flows Invertibility

I am currently reading up on RealNVP, which has the following transformations according Lilian Weng: $$ \begin{aligned} \mathbf{y}_{1:d} &= \mathbf{x}_{1:d} \\ \mathbf{y}_{d+1:D} &= \mathbf{x}...
user2793618's user avatar
10 votes
6 answers
2k views

Why do Denoising Diffusion Probabilistic Models (DDPM) add noise according to $\sigma_t$ during sampling?

Reading about Denoising Diffusion Probabilistic Models (DDPM) the paper (algorithm 2 - sampling) states that the sampling goes according to... $$ x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{...
Joff's user avatar
  • 942
2 votes
1 answer
1k views

How to generate new data with a VAE?

I have built the following function which takes as input some data and runs a VAE on them: ...
quant's user avatar
  • 521
4 votes
1 answer
947 views

How to rewrite DreamBooth loss in terms of $\epsilon$-prediction?

I'm trying to make the loss used in DreamBooth paper explicit, writing it in terms of the noise, as it is commonly written in the original diffusion article [1], instead of the image reconstruction ...
Ciodar's user avatar
  • 495
3 votes
0 answers
408 views

If neural networks are deterministic once trained how are generative able to generate different responses to inputs?

Once training is finished and a neural network is in inference mode, its outputs will be deterministic, i.e. the weights have been fixed. How is it then that generative models are able to generate non-...
Jose Miguel Cruz y Celis's user avatar
1 vote
1 answer
61 views

Bayes theorem in the context of generative classification models

In the Introduction to Statistical Learning p. 142 in chapter 4.4 on generative models for classification the formula $P(Y = k|X = x) = \frac{π_k \cdot f_k (x)} {\sum_{l=1}^{K}π_l f_l(x)}$ is given to ...
LololoTheGreat's user avatar
12 votes
1 answer
2k views

Purpose of scaling mean by $\sqrt{1 - \beta_t}$ in forward diffusion process

In the forward diffusion process described by Ho, et al. the probability distribution for the next step is: $$q(\mathbf{x}_t|\mathbf{x}_{t-1}) = N(\mathbf{x}_t;\sqrt{1-\beta_t}\mathbf{x}_{t-1},\beta_t\...
Adrian Stoll's user avatar
0 votes
0 answers
46 views

Adjusting weight updates in a generative model

Suppose I am training a generative model G to produce vectors z in R^d, where d is fixed. The objective of G is to produce realistic vectors, which I am calling the "reality objective". ...
postnubilaphoebus's user avatar
3 votes
2 answers
681 views

How is the variance for a diffusion kernel derived for a diffusion model?

So I'm watching this video tutorial from CVPR this year on diffusion models, and I am confused by the variance term in the distribution on the left on the video. I understand that in the forward ...
Cynthia Kim's user avatar
1 vote
2 answers
429 views

Deriving the Reparameterization Trick

I am reading about the reparameterization trick from here. It states $\boldsymbol{\epsilon}\sim p(\boldsymbol{\epsilon})$, $\textbf{z}=g_\theta(\boldsymbol{\epsilon},\textbf{x})$, and $$\mathbb{E}_{p_\...
muser's user avatar
  • 145

1
2 3 4 5 6