Questions tagged [generative-models]
A probabilistic or statistical model thought about as describing how the values in a sample is actually generated, and not only as a description or approximation.
272
questions
1
vote
1
answer
32
views
Rewrite linear expression
Is it possible to rewrite
$$
\frac{-1}{2}\left(x^T\Gamma^{-1}(\mu_1-\mu_0)+(\mu_1-\mu_0)^T\Gamma^{-1}x\right)
$$
as
$$
-\theta^Tx
$$
where $x,\mu\in\mathbb{R}^d, \Gamma\in\mathbb{R}^{d\times d}$ and $\...
- 13
0
votes
0
answers
21
views
Normalizing Flows Invertibility
I am currently reading up on RealNVP, which has the following transformations according Lilian Weng:
$$
\begin{aligned}
\mathbf{y}_{1:d} &= \mathbf{x}_{1:d} \\
\mathbf{y}_{d+1:D} &= \mathbf{x}...
- 17
0
votes
0
answers
7
views
Classification via "Activation Maximization"
I came across the following problem and I'm unsure if a solution exists or if it has been proven that what I'm trying to achieve is impossible. I tried to google for papers on the topic but I haven't ...
- 101
2
votes
0
answers
76
views
Why do Denoising Diffusion Probabilistic Models (DDPM) add noise according to $\sigma_t$ during sampling?
Reading about Denoising Diffusion Probabilistic Models (DDPM) the paper (algorithm 2 - sampling) states that the sampling goes according to...
$$
x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{...
- 700
0
votes
0
answers
13
views
Output of EBM energy network
I implemented an energy-based model (EBM) as the prior of a generative model. The energy network is a small multi-layer perceptron with parameters $\mathbf{\theta}$. The prior probability is given as
$...
- 1
1
vote
1
answer
98
views
How to generate new data with a VAE?
I have built the following function which takes as input some data and runs a VAE on them:
...
- 431
2
votes
0
answers
269
views
How to rewrite DreamBooth loss in terms of $\epsilon$-prediction?
I'm trying to make the loss used in DreamBooth paper explicit, writing it in terms of the noise, as it is commonly written in the original diffusion article [1], instead of the image reconstruction ...
- 358
0
votes
0
answers
9
views
SEQGAN model strange behaviour
I have been trying SEQGAN to increase the size of my small dataset with around 5000 observations. I used the the defaults settings so far and the model looked like underfit because the generated ...
- 11
3
votes
0
answers
49
views
If neural networks are deterministic once trained how are generative able to generate different responses to inputs?
Once training is finished and a neural network is in inference mode, its outputs will be deterministic, i.e. the weights have been fixed. How is it then that generative models are able to generate non-...
1
vote
1
answer
48
views
Bayes theorem in the context of generative classification models
In the Introduction to Statistical Learning p. 142 in chapter 4.4 on generative models for classification the formula
$P(Y = k|X = x) = \frac{π_k \cdot f_k (x)} {\sum_{l=1}^{K}π_l f_l(x)}$ is given to ...
2
votes
1
answer
114
views
Purpose of scaling mean by $\sqrt{1 - \beta_t}$ in forward diffusion process
In the forward diffusion process described by Ho, et al. the probability distribution for the next step is:
$$q(\mathbf{x}_t|\mathbf{x}_{t-1}) = N(\mathbf{x}_t;\sqrt{1-\beta_t}\mathbf{x}_{t-1},\beta_t\...
- 23
0
votes
0
answers
22
views
Adjusting weight updates in a generative model
Suppose I am training a generative model G to produce vectors z in R^d, where d is fixed. The objective of G is to produce realistic vectors, which I am calling the "reality objective". ...
1
vote
1
answer
61
views
How is the variance for a diffusion kernel derived for a diffusion model?
So I'm watching this video tutorial from CVPR this year on diffusion models, and I am confused by the variance term in the distribution on the left on the video. I understand that in the forward ...
- 11
0
votes
0
answers
14
views
How is validity of generative model associated with optimizing lower bound?
When reading the paper here, in the Related Works section, I see a statement "Although optimising a lower bound is not strictly necessary for disentangling, it does ensure that we have a valid ...
- 401
0
votes
1
answer
72
views
Deriving the Reparameterization Trick
I am reading about the reparameterization trick from here.
It states
$\boldsymbol{\epsilon}\sim p(\boldsymbol{\epsilon})$, $\textbf{z}=g_\theta(\boldsymbol{\epsilon},\textbf{x})$, and
$$\mathbb{E}_{p_\...
- 141
2
votes
2
answers
214
views
Can the dimension of the latent space in VAEs, be larger than the dimension of the data?
I am experimenting with VAEs. There, there is a parameter that you need pass when you create the NN, which is the dimension of the latent space.
In the typical ...
- 431
0
votes
0
answers
24
views
Draw conditional samples from GMM
I have a dataset with 4 numerical features and class labels for each data point. I have implemented a GMM classifier to predict the class of each training sample.
Now I would like to generate new data ...
- 1
1
vote
1
answer
43
views
VAE mean and Standard Deviations are input dependent?
The original presentation of variational autoencoders, VAE assumes the mean $\mu$ and the sd $\sigma$ are functions of the input variable, say $x$. I am studying "Learning Structured Output ...
- 13
0
votes
0
answers
13
views
How training GANs with the method of optimizing 2 loss functions simultaneously differ from the actual alternate training?
I am training GAN in a multiobjective optimization setting where I am optimizing both the loss functions(generator and discriminator) at the same like optimizing 2 functions simultaneously. However, ...
- 109
0
votes
0
answers
10
views
Behaviour of Deep Generative Models on non-Gaussian toy data
I was wondering how vanilla Deep Generative Models, e.g. (W)GANs or VAEs are expected to behave on standard toy non-Gaussian benchmark datesets of the likes of two-moons or circles. Is there any ...
- 924
1
vote
2
answers
97
views
Autoregressive models for modeling conditional probabilities
I'm reading Tomczak's Deep Generative Modeling. When the author discusses auto-regressive models, he mentions that we model the probability distribution $p(\mathbf{x})$ of the data $\mathbf{x}$ as $$p(...
1
vote
1
answer
45
views
Understanding conditional probability formulas in the context of class-conditionals in generative models
I am trying to understand the theory behind probabilistic generative models a bit better.
If I model the class-conditionals as Gaussians, the formula is this:
$$
\frac{1}{2\pi^{\frac{D}{2}}|\Sigma|^...
- 219
0
votes
0
answers
58
views
In a WGAN, how do I train Discriminator more than the generator?
I am working with a WGAN using PyTorch and I came across a training option that you must train the discriminator for more iterations than the generator in WGAN-GP model. So this is the following code ...
- 111
0
votes
0
answers
25
views
Why the quadratic term is cancelled for general K>2 classes in probabilistic generative models?
The problem comes from a paragraph in section 4.2.1 of Christopher M. Bishop's book "Pattern Recognition and Machine Learning". This is a paragraph that deals with general $K>2$ classes ...
- 183
0
votes
0
answers
98
views
What's GAN's Input-size Limitations?
I am interested in GAN for generating synthetic data. I am studying the input limitations for GAN starting from which GAN is no longer usable.
I have found many applications that use GANs for ...
- 101
1
vote
0
answers
42
views
How to implement simple VAE with sparse tensor in Tensorflow
thank you for reading.
I have been attempting to train a simple VAE on very sparse 2D and 3D data. So far I have been training using dense tensors which - I think - is resulting in horrible training ...
- 11
1
vote
0
answers
49
views
Can you make a simple 2D ring with a GAN?
I am trying to model simple 2d continuous distributions with GANs. Here, I focus on a 2d distribution following a ring structure. The architecture of my networks are:
...
- 133
0
votes
0
answers
35
views
Better noise distribution in GANs?
I am just wondering if there is something analogous to the Kaiming He Initialization in GANs for better training (better convergence, training time, etc.)? For example, can the generative model use ...
1
vote
1
answer
47
views
Can posterior become tractable if we know p(x)?
In the VAE framework where x is an input data (a vector) and z is a vector of continuous latent variables,
the posterior ...
2
votes
1
answer
45
views
Parametric vs non-parametric generative models
I have a little perplexity trying to distinguish parametric vs non-parametric generative model.
In my understanding, a parametric generative model would try to learn the probability density function ...
- 537
0
votes
0
answers
42
views
What is the learning procedure and purpose of class embeddings in Conditional GAN's
I was learning about conditional GAN's, and there is a point I did not understand about the training process. To specify the class we want to generate images from, we specify a number, which is then ...
- 145
0
votes
1
answer
196
views
What is the exact role of model $p_\theta$ in Diffusion models for the reverse process?
I'm reading this interesting blog post explaining Diffusion probabilistic models and trying to understand the following.
In order to compute the reverse process, we need to consider the posterior ...
- 537
5
votes
2
answers
1k
views
Why Logistic Regression is not a generative model?
I was reading about the difference between discriminative and generative models, and I read that Discriminative models learn only the boundary between classes hence they are not able to to create new ...
- 145
1
vote
1
answer
332
views
How to reconstruct a Euclidean distance matrix from grouped pairwise-distance means and standard deviations?
Coordinates and Labels
Take the simple case of 3 distinct object classes and 5 instances of each class situated in 3D Euclidean space. The coordinates and labels might look like the following:
...
- 69
1
vote
0
answers
21
views
How does the fixed point interation in invertible resnets work?
I feel like I am missing some easy point about this invertible resnet paper which is making it hard for me to grasp how the fixed point iteration works.
stated simply, the residual connection in a ...
- 700
1
vote
0
answers
205
views
Classical VAE not learning 2D gaussian mixture distribution using MSE loss
I've been exploring VAE for non-image data. I consider small to medium-sized continuous vector spaces and I want to learn the distribution of a dataset in that space.
As a warm up exercise, I tried ...
- 113
1
vote
1
answer
33
views
Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?
I've noticed in several resources on variational autoencoders (for example the Wikipedia article), we use the same parameters theta ($\theta$) for the prior, likelihood, posterior, etc distributions. ...
- 11
0
votes
0
answers
27
views
Techniques Used for DeepFake and Its Corresponding Research Field
I'm a beginner in image generative models, I'm trying to do some work similar to DeepFake, therefore I would like to find out first what techniques DeepFake use to generate the fake videos. Do they ...
- 1,813
1
vote
1
answer
304
views
Does a CNN always learn a latent space?
In general, a latent space is a structure of reduced dimensionality than that of the input space where points on this space share resemblance the closer they are to each other.
This article also ...
- 659
0
votes
0
answers
54
views
Using Inception and FID scores in training?
Is it possible to use the Inception and FID scores in the training of a deep image generation model, i.e. to maximize the scores in a loss function, albeit this is "cheating"? If so, has ...
- 1
1
vote
0
answers
143
views
GANs - why does the generator want to minimize the loss (intuition)
I am a little bit puzzled about the following. In a generative adversarial network, we consider a binary classification problem with a binary cross-entropy loss.
Now, the generator wants to minimize ...
- 111
3
votes
1
answer
1k
views
Should I be using batchnorm and/or dropout in a VAE or GAN?
I am trying to design some generative NN models on datasets of RGB images and was debating on whether I should be using dropout and/or batch norm.
Here are my thoughts (I may be completely wrong):
...
- 197
1
vote
1
answer
418
views
Initialization of GAN discriminator
The question is pretty straightforward: how are GAN and WGAN discriminators typically initialized?
I couldn't find much info on this. E.x. for GANs, I imagine you would theoretically want the ...
- 133
2
votes
0
answers
76
views
Restricted Boltzmann Machine: W matrix visualization results after training MNIST images and Pseudo-log-likelihood
I am implementing RBM from scratch using Tensorflow and after training my RBM on the MNIST dataset for 200 epochs using Persistent CD with two steps of contrastive divergence, I learn the weights W ...
- 21
1
vote
0
answers
148
views
How to estimate joint probability or conditional probability using marginal ones?
I have 2 datasets:
The 1st one gives us the probability that $m$ events occur on $n$ observations ($m$ columns for $n$ rows)
The 2nd one tells us if the event occurred (1 if occurred, 0 else ; always ...
- 11
0
votes
0
answers
98
views
A Generative model for binary classification (Modeling, Network architecture
I'm trying to build a network to classify input $X$ into 2 categories with a generative manner. This is because there is harsh imbalance issue in data.
But I can't understand how to train the model.
...
- 1
0
votes
0
answers
54
views
Problem for training Wasserstein GAN
I'm trying to train a Wasserstein GAN to guess sparse one-hot encoded matrices (0/1), in particular I've reimplemented the same architecture proposed in this paper.
The problem, as you can see, is ...
- 537
0
votes
1
answer
37
views
Is there such a thing as intra-sample modal collapse in GANs?
Mode collapse is a known issue in generative adversarial networks (GANs) whereby the generator only learns a subset of the real data distribution. In those cases, it only outputs variations of a small ...
- 169
1
vote
0
answers
115
views
Synthetic data generation - GANs vs Simulator?
For synthetic data generation, does the GAN perform better than a simulator? If so, what are the limitations of the simulator?
If we consider Conditional GANs, we could generate data based on the ...
- 111
0
votes
1
answer
104
views
Computing a prior from two components in Naive Bayes
Given a model parameter $\theta$ that is composed of two distributions in a Naive Bayes classifier, how is $P(\theta)$ typically computed in practice?
More specifically, from the article of Nigam et ...
- 240