Questions tagged [generative-models]

A probabilistic or statistical model thought about as describing how the values in a sample is actually generated, and not only as a description or approximation.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

what is the number of bits assigned by a generative model to a K-class dataset, in theory

Consider a dataset, something like MNIST or CIFAR-10, that consists of (for example) 50000 images taken from 10 classes and balanced with an equal number in each class. For a discriminitive classifier,...
0
votes
0answers
13 views

Is Beta-VAE regularization equivalent to using a Gaussian prior with variance smaller than 1?

The Beta-VAE model offers the regularization over the -$\beta$ KL(q(z|x)||p(z)) term of the ELBO term in VAE formulation. Scaling this term up forces some latents to be distributed as unit Gaussians (...
1
vote
1answer
38 views

How does fitting a generative model ($P(X,Y)$) allow one to generate examples $(X,Y)$?

For example, suppose I have a database of images of cats ($C$) and dogs ($D$). My database of labeled images consists of $(X,Y)$ where $X$ is a pixelated image and $Y \in \{C,D\}$. If I somehow fit a ...
0
votes
0answers
25 views

Why are energy-based models still typically trained with likelihood?

Lately I've been getting into Energy-based models through some of Yann LeCun's talks where he advocates the use of non-normalized models because it allows for more flexibility in the choice of loss ...
0
votes
0answers
11 views

How to train a deep learning model to choose K (variable number) elements from a set of N elements?

In my use case, I am able to generate sets with a fixed number of elements (N) using a GAN. However, the underlying distribution of the sets has a variable number of elements. Now I need the model to ...
2
votes
0answers
30 views

For a generative model, how is modelling p(X,Y) equivalent to modelling P(X|Y=y)?

On the Wikipedia page for generative models it gives the following definitions of a generative model: (X is an observable variable, Y is the target variable) 1) A generative model is a model of the ...
0
votes
0answers
19 views

Are word2vec, contrastive predictive coding, etc. examples of “energy based models”?

I am trying to place contrastive learning models, such as word2vec and Contrastive Predictive Coding [1] in the context of other generative models, such as Autoregressive Models, VAEs, GANs, and ...
0
votes
0answers
20 views

Why loss in WGAN_GP fluctuates so much?

I am trying to implement the Improved Training of Wassertein GANs paper. Actually I am simply trying to convert the authors TF code to PyTorch (TF is giving some dependence issues and I am much-much ...
2
votes
1answer
113 views

Normalizing flow training

I've been learning about normalizing flow. This is my understanding, and please correct me whenever I am wrong. There are $\{y_1,y_2,...,y_n\}$ samples from an unknown distribution $p_y(y)$ that we ...
1
vote
1answer
29 views

Understanding step in proof of GAN algorithm convergence, involving convexity

I am reading the original paper on GANs, https://arxiv.org/abs/1406.2661. The proof of proposition 2, on the convergence of the gradient descent algorithm reads Consider $V(G, D) = U(p_g, D)$ as a ...
1
vote
2answers
82 views

What concept comes before VAE and after GMM?

Suppose I am designing a course on generative models and I have just finished discussing GMM. My goal is to teach VAE. However, VAE's technicality is very high. Does there exist some model in between ...
1
vote
0answers
31 views

Normalizing Flows notations and change of variable formula

Maybe it is a silly question, but I am wondering why in some papers the function $f$ is the mapping from $x$ to $z$ and in some others it is the other way around. Typically in this review we go from $...
2
votes
1answer
26 views

Can a Simple ANN be Generative?

If a simple ANN was trained to predict the next step in a sequence, such as a univariate time series, can it be considered a generative model?
2
votes
1answer
112 views

Are text generation models generative or discriminative?

I've recently been studying generative and discriminative models, and I had a question regarding text generation. I'm aware that generative models model $P(X, Y)$ and discriminative models model $P(Y |...
0
votes
1answer
106 views

How to find bits/dim of a gaussian output distribution?

I have images that are 64x64x3 and 64x64x1 8-bit. I transform those images down to [-1,1]. I now want to find the bits/dim for my VAE log probability. How do I find the bits/dim of the log likelihood? ...
2
votes
1answer
10 views

Updating of characer's embedding inside an RNN during character generation task

In one of the tutorials of tensorflow, there is a "text generation with an RNN" tutorial. When creating the model, they create a mapping of characters to IDs and vice versa. Then in the ...
0
votes
0answers
10 views

Difference between Structural Topic Modeling(STM) and SAGE (Sparse Additive Generative Model)?

I have read that STM combines 3 models of: (1) correlated topic model (CTM) (2) Dirichlet-Multinomial Regression (DMR) topic model (3) Sparse Additive Generative Model (SAGE) Is it correct to just ...
1
vote
1answer
119 views

When is my Wasserstein GAN-GP overfitting?

I have a hard time interpreting the WGAN-GP losses attached. At which epoch is D and/or G overfitting? The quality improves a lot overtime, yet the generator loss at later epochs does not appear on ...
0
votes
0answers
22 views

Projection pursuit optimal hidden units RBM

I am training a Restricted Boltzmann machine. I gather from this presentation: https://cseweb.ucsd.edu/~dasgupta/254-deep/nakul.pdf that projection pursuit can be used for density estimation of the ...
0
votes
0answers
25 views

Why does using conditional random field avoid independence assumption

I am reading about conditional random fields in Daphne Koller's book on probabilistic graphical models. One of the advantages to using CRF is that we can avoid modelling the correlations between ...
0
votes
0answers
35 views

Wrong reconstruction with Wasserstein loss conditional GAN with gradient penalization (WcGAN-GP)

I have a following dataset of $(X_\mathrm{real}, y_\mathrm{real})$ sample/label pairs, where both $X_\mathrm{real}$ and $y_\mathrm{real}$ are multidimensional vectors. I'm trying to build the ...
1
vote
1answer
57 views

Latent Dirichlet Allocation - dimensionality of the Dirichlet prior parameter

I seeking some clarity on the dimensionality of the (hyper)parameter $\eta$ of the "smoothed LDA" model in Section 5.4 of the original paper by Blei, Ng, Jordan (2003), which can be found ...
0
votes
0answers
7 views

Network structure of generative model for classification

I'm trying to model a generative model for classification problem, especially aiming to solve an imbalanced data problem. However, I couldn't get intuitive understanding for generative classifier in ...
0
votes
0answers
58 views

Is it wrong to use tanh with images normalized in [0,1] range?

I've seen in some repositories, mostly related to GANs (Generative Adversarial Networks) using tanh activation function whilst having input images in the range of <...
-1
votes
1answer
26 views

K-Lipschitz for the Wasserstein GANs

I am trying to follow this blog for Wasserstein loss for Generative Adversarial Networks: From GAN to WGAN. Actually, I am trying to follow the logic behind the K-Lipschitz continuity. This post in ...
2
votes
1answer
695 views

Why use tanh function at the last layer of generator in GAN?

While studying GAN, I found out that ReLU activation is used at the intermediate layers, and tanh or sigmoid is used at the last layer of the generator. I'm curious about why sigmoid or tanh is used ...
0
votes
0answers
12 views

Help in calculating diagonal covariance matrix for generative model for binary classification

I am given this data. I want to fit a generative model $\cal{N}(\mu_0, \sigma_0^2 I_2)$, $\cal{N}(\mu_1, \sigma_1^2 I_2)$ for the $0$ and $1$ classes respectively using $\textbf{MLE}$ and plot ...
0
votes
1answer
33 views

How do I revert a probability distribution back to its source data samples?

Normally we first collect real samples into a dataset and describe its probability distribution parametrically or empirically. If I instead generate a parametric distribution for artificial data ...
1
vote
1answer
72 views

Modern applications of Bayesian Model Selection

I'm trying to understand the merits of this field so I'll try to break down my question. Research: Is Bayesian model selection considered a popular topic of research these days? Variable selection: ...
1
vote
0answers
18 views

entropy regularization in generative model

I am wondering if it is possible to use entropy as a regularization in a generative model. For example, in the conjugate model where $x_i \in X$ is observed data and generated from a Normal ...
2
votes
1answer
108 views

Inference in Normalizing Flow model: NICE(non linear independent components estimation)

I was recently reading Y. Bengio's paper on NICE (https://arxiv.org/abs/1410.8516). In the paper, authors have taken a view that a good representation involves easy learning of the data distribution. ...
2
votes
0answers
56 views

Can PCA generate a new random image?

I read The Batch: GANs newsletter and Goodfellow said: My colleague Bing Xu modeled face images from the Toronto Face Database, which were only 90 pixels square and grayscale. Because the faces were ...
1
vote
0answers
40 views

Probabilistic Difference between Autoencoders and Variational Autoencoders

I have recently read up about Autoencoders and Variational Autoencoders. In Variational Autoencoders, the loss is modeled based on what distribution we choose for P(x|z). So, if we choose it to be ...
0
votes
0answers
19 views

Probabilistic generative models for clustering and classification

I have a question regarding the probabilistic setting of clustering and classification. More specifically regarding Gaussian Mixture Models and probabilistic generative models for classification. In ...
0
votes
1answer
136 views

Why VAE need reparameterization trick while LDA does not (both using variational inference for optimization)

Both VAE and LDA (latent Dirichlet allocation) is based on variational inference, and they both try to optimize ELBO objective function Variational autoencoders use reparameterization so that "...
0
votes
0answers
47 views

Variational Autoencoder with a Flexible Prior

Let's say I have a Variational autoencoder with a Gaussian prior architecture and I want to regularize this VAE with a flexible prior. Does sampling with the normal "reparameterization trick"...
5
votes
1answer
517 views

How well does GAN (generative adversarial network) perform for small samples?

GAN is an unsupervised learning algorithm that pits a discriminator and generator against one another so that they iteratively compete to enhance the overall model's ability to model/replicate a given ...
0
votes
2answers
342 views

In VAE, why use MSE loss between input x and decoded sample x' from latent distribution?

Variational Autoencoders (VAEs) are based on the concept of Variational Inference (VI) and use two Neural Networks similar to Vanilla Autoencoders (AEs) for function approximation. I understood the ...
2
votes
1answer
506 views

Why we learn $\log{\sigma^2}$ in VAE reparameterization trick instead of standard deviation?

We know that the reparameterization trick is to learn two vectors $\sigma$ and $\mu$, sample $\epsilon$ from $N(0, 1)$ and then your latent vector $Z$ would be (where $\odot$ is the element-wise ...
5
votes
1answer
234 views

reparameterization trick in VAEs, How should we do this?

I'm confused about how does reparameterization trick works. In this article shows it very simple. You learn two vectors $\sigma$ and $\mu$, sample $\epsilon$ from $N(0, 1)$ and then your latent ...
4
votes
2answers
319 views

Variational Autoencoder (VAE) latent features

I'm new to DL and I'm working on VAE for biomedical images. I need to extract relevant features from ct scan. So I created first an autoencoder and after a VAE. My doubt is that I don't know from ...
0
votes
1answer
358 views

What is the Intuition behind the GAN Discriminator loss? How does Discriminator loss works?

I have just stated learning GAN and the loss used are different for same problems in same tutorial. Could someone please tell me intutively that which loss function is doing what? For example, in the ...
1
vote
2answers
1k views

What is the stop criteria of generative adversarial nets?

I have used the GANs (Generative Adversarial Networks) with a binary cross-entropy loss function (in both generator and discriminator). Throughout the training step, the variation of generator loss ...
0
votes
0answers
28 views

Conditional generative models for large numbers of classes

I'm looking for a conditional generative model — could be cGAN or cVAE — that is particularly well suited to large numbers of classes. I'm expecting to require 500+ classes, but so many models are ...
2
votes
1answer
391 views

Generating distributions with with given variance, skewness, and kurtosis

I would like to generate distributions that are as close to normal as possible, except for the deviations shows below. The options I've located have properties that are related to the families of ...
0
votes
1answer
22 views

Generating text using LSTM given condition vector

I know that you can use an RNN to generate text given the first few letters ...
0
votes
0answers
7 views

the IS(Inception Score) of the generated images is higher than the images from datasets?

Is there a situation where the IS(Inception Score) of the generated images is higher than the images from datasets? Can anyone tell me?
2
votes
0answers
20 views

Softmax classifier with class priors

Softmax classifier is a discriminative model that directly models $p(Y|X, w)$ where $Y$ is the label for input $X$. We can write it as follows: $$p(y_i=k|x_i,w_k)=\frac{\exp \lbrace w_k^T x_i \rbrace}{...
2
votes
0answers
32 views

Extremely basic question: how are data assumed to be generated in machine learning?

Given a dataset $\mathcal{D} = \{x_i\}, i = 1, \ldots, N, x_i \in \mathbb{R}$ In machine learning, what assumption is made as to how data are generated? I've seen two basic ideas circulating around, ...
1
vote
1answer
17 views

How do we generate the samples of hidden root nodes in the Bayes network (Sigmoid Belief Networks) of a generative model

Following is a Sigmoid Belief Networks where we can only observe the bottom observable layer $v.$ Usually we use ...

1
2 3 4 5