Questions tagged [generative-models]

A probabilistic or statistical model thought about as describing how the values in a sample is actually generated, and not only as a description or approximation.

109 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7 votes
0 answers
718 views

Is google's wavenet architecture computing a bunch of values that it will never use?

I've been trying to understand the wavenet paper. In order to do so, I am using this implementation that I found on github because it gets good results and it is pretty clear. But I have a question ...
Andrés Marafioti's user avatar
5 votes
0 answers
376 views

What enforces features diversity in RBM?

I'm working on an implementation of a Restricted Boltzman Machine (RBM). I made some tests on the MNIST dataset trying to learn a representation of the digit 2. My inputs are binary images. My aim is ...
user3091275's user avatar
4 votes
0 answers
318 views

Are there nonparametric generative models for datasets?

Typically when I see generative models, e.g., Latent Dirichlet Allocation (JMLR) or Linear/Quadratic Discriminant Analysis (wikipedia LDA), they are probabilistic models that belong to the exponential ...
Sleepy 17's user avatar
4 votes
0 answers
2k views

GANs for non image data

I'm looking to narrow down the subject for my bachelor thesis: I am currently working on a project, that only offers a small dataset and there will be no more data incoming for now. What I'm trying to ...
nicenoize's user avatar
4 votes
0 answers
3k views

How is the Concept of masked convolutions in PixelRNN implemented?

I am reading the PixelRNN paper (https://arxiv.org/abs/1601.06759). I am having a little trouble understanding how one implements the concept of masked convolutions. Section 3.4 states: "The masks ...
Sahil's user avatar
  • 141
3 votes
0 answers
327 views

If neural networks are deterministic once trained how are generative able to generate different responses to inputs?

Once training is finished and a neural network is in inference mode, its outputs will be deterministic, i.e. the weights have been fixed. How is it then that generative models are able to generate non-...
Jose Miguel Cruz y Celis's user avatar
3 votes
0 answers
183 views

For a generative model, how is modelling p(X,Y) equivalent to modelling P(X|Y=y)?

On the Wikipedia page for generative models it gives the following definitions of a generative model: (X is an observable variable, Y is the target variable) 1) A generative model is a model of the ...
apprentice9's user avatar
3 votes
0 answers
84 views

entropy regularization in generative model

I am wondering if it is possible to use entropy as a regularization in a generative model. For example, in the conjugate model where $x_i \in X$ is observed data and generated from a Normal ...
JYY's user avatar
  • 737
3 votes
1 answer
429 views

Can someone explain CAM loss used in U-GAT-IT paper?

I have been reading a recent paper accepted at ICLR, U-GAT-IT, which seems to produce pleasing results in the image-to-image translation tasks. There are four kinds of loss used in this paper: ...
Sudarshan Regmi's user avatar
3 votes
1 answer
71 views

Why are autoregressive models neither directed or undirected, as described in the NADE paper?

In the paper Neural Autoregressive Distribution Estimation (Uria et al., 2016), NADE (and other autoregressive models) seem to be described as neither directed or undirected models: We’ve described ...
Christabella Irwanto's user avatar
3 votes
0 answers
607 views

How to use KL divergence to compare two distributions?

I am trying to model the probability distribution of a multi-dimensional dataset where all the values are discrete. Suppose the training data (represented by T) is of the shape (m, n) where n is the ...
Random Person10's user avatar
3 votes
0 answers
97 views

Generative Adversarial Network and Variational Autoencoders for Independent Component Analysis?

Background: I'm working on a model for independent component analysis (ICA) that is based on a methodology similar to GANs and VAEs. What I'm having trouble understanding is how the choice of the loss ...
Don Walpola's user avatar
  • 1,328
3 votes
0 answers
285 views

Generative Adversarial Networks - Gradient saturation

This is the value function from the GANs paper: The authors explain that this equation "may not provide sufficient gradient for $G$ to learn well", because early in the learning process the ...
cmplx96's user avatar
  • 463
3 votes
1 answer
66 views

Is there some theory of SVMs with infinitely many data?

I am trying to understand what does it means to have a (linear) SVM classifier (with soft margins) given the generative model of the data. And I realize I have not seen any paper on it, nor can I ...
Jacques Wainer's user avatar
3 votes
0 answers
59 views

Generative modelling: what if the generating models have very different "quality of fit"

Say I want to classify my data into two categories. I am pretty sure that my data has been generated by two mixtures of Gaussians -- on has a bimodal and one a trimodal form. I then train the ...
bayerj's user avatar
  • 13.7k
2 votes
1 answer
426 views

How is the variance for a diffusion kernel derived for a diffusion model?

So I'm watching this video tutorial from CVPR this year on diffusion models, and I am confused by the variance term in the distribution on the left on the video. I understand that in the forward ...
Cynthia Kim's user avatar
2 votes
1 answer
874 views

What is the exact role of model $p_\theta$ in Diffusion models for the reverse process?

I'm reading this interesting blog post explaining Diffusion probabilistic models and trying to understand the following. In order to compute the reverse process, we need to consider the posterior ...
James Arten's user avatar
2 votes
0 answers
97 views

Restricted Boltzmann Machine: W matrix visualization results after training MNIST images and Pseudo-log-likelihood

I am implementing RBM from scratch using Tensorflow and after training my RBM on the MNIST dataset for 200 epochs using Persistent CD with two steps of contrastive divergence, I learn the weights W ...
ef24's user avatar
  • 21
2 votes
0 answers
191 views

What is the "lower bound average gain" metric used in GMM stopping criterion used in Scikit learn?

In Scikit Learn's GMM class, it says that GMM training algorithm stops according to the "lower bound average gain" https://scikit-learn.org/stable/modules/generated/sklearn.mixture....
Fraïssé's user avatar
  • 1,520
2 votes
0 answers
391 views

Can PCA generate a new random image?

I read The Batch: GANs newsletter and Goodfellow said: My colleague Bing Xu modeled face images from the Toronto Face Database, which were only 90 pixels square and grayscale. Because the faces were ...
Thew's user avatar
  • 121
2 votes
0 answers
29 views

Softmax classifier with class priors

Softmax classifier is a discriminative model that directly models $p(Y|X, w)$ where $Y$ is the label for input $X$. We can write it as follows: $$p(y_i=k|x_i,w_k)=\frac{\exp \lbrace w_k^T x_i \rbrace}{...
groove's user avatar
  • 473
2 votes
0 answers
35 views

Extremely basic question: how are data assumed to be generated in machine learning?

Given a dataset $\mathcal{D} = \{x_i\}, i = 1, \ldots, N, x_i \in \mathbb{R}$ In machine learning, what assumption is made as to how data are generated? I've seen two basic ideas circulating around, ...
Fraïssé's user avatar
  • 1,520
2 votes
1 answer
544 views

How do I resize output of a generative network to reconcile with the size of the real data?

I am designing a Generative Adversarial Network (GAN) trained on an image dataset. It has two components: the generator and the discriminator. The generative network outputs an artificial image. The ...
hazrmard's user avatar
  • 208
2 votes
1 answer
327 views

Controlling details in images generated by Generative Adversarial Networks (GANs)

With conditional GANs it is possible to generate images of a certain class of objects. And moreover with current text-to-image methods it seems to be possible to control certain details of the ...
mrvn's user avatar
  • 21
2 votes
1 answer
143 views

How to test generated text

I am creating a text generation algorithm for my master's research. I have a dialogue between two people and I would like to simulate one part of the conversation with naturally generated text (not ...
Bennie van Eeden's user avatar
2 votes
0 answers
25 views

Generating distributions with very small values with generative adverserial networks

I am using a GAN to generate a vector distribution. However, most of the values of this distribution are very close to zero, i.e $1 \times 10^{-7}$. The distribution also have minus values. With ...
SameeraR's user avatar
  • 121
2 votes
0 answers
53 views

Extracting likelihoods from generative model

I am looking for papers dealing with the extraction of explicit descriptions of probability distributions from a generative model. My use case is the following: I trained a GAN to generate samples ...
lbf_1994's user avatar
  • 528
2 votes
0 answers
48 views

How do I check my GAN implementation is correct?

I wrote a GAN implementation and I trained that to produce some sample images after training on a dataset. The images looked visually fine. Now I want to test my implementation on the CI and make ...
abc's user avatar
  • 121
2 votes
0 answers
69 views

Understanding probabilistic inference graphs

I am having trouble understanding inference graphs. In the diagram below I understand the graph on the left (forward graph) where the arrows describe the direction that data flows when training for ...
piccolo's user avatar
  • 937
2 votes
0 answers
125 views

What is the difference between a conditional model and just having multiple models?

Say I have a labeled dataset that I want to create a generative model for, like a Generative Adversarial Network or a Variational Autoencoder. What do I gain or lose by making my models conditional (i....
Harm van den Brand's user avatar
2 votes
0 answers
84 views

GP-WGANs with Minibatch Discrimination

In the "Improved Training of Wasserstein GANs" paper the authors mentioned that batch normalization can not be used in combination with gradient penalty, since it introduces correlation between ...
OwlOfAthena's user avatar
2 votes
0 answers
37 views

In paper “Neighbourhood Components Analysis”, how to determine the optimal number of neighbours (K)?

Recently, I am reading the paper "Neighbourhood Components Analysis" (Goldberger, Jacob, et al. "Neighbourhood components analysis." Advances in neural information processing systems. 2005.). At the ...
Tengerye's user avatar
  • 201
2 votes
0 answers
672 views

Training Wavenet on time series data

I am trying to train a predictive model on EEG signals, because of the high frequency of signals I am using a wavenet, after training when I use it as a generative model (like PixelCNN) it fails ...
Separius's user avatar
  • 121
2 votes
0 answers
509 views

Generative Network Sampling: spherical interpolation instead of linear?

I am building my own implementation of a DCGAN at the moment and want to try different random samplings for the Generator's input (noise). Recently, I found the paper Sampling Generative Networks. I ...
daniel451's user avatar
  • 2,915
2 votes
0 answers
112 views

Beginner working with generative models

I'm getting a set of 20-dimensional feature vectors. I'm getting a 100 vectors per second. These feature vectors are occuring in a sequence, where one feature vector depends on the one that came ...
arkate's user avatar
  • 121
2 votes
0 answers
1k views

Difference between a generative MRF and discriminative CRF

I am having trouble developing the intuition behind the difference between a regular generative Markov random field (MRF) and its discriminative counterpart. So, as I think I have understood so far ...
Luca's user avatar
  • 4,650
2 votes
0 answers
91 views

Treating multiple Dichotomies combined?

Let's say I am interested in choosing a new country $c_1, \ldots, c_k$ to live in. For some reason I can only apply to one country and only once. I know for each country a set of 2000 observations (...
Chris's user avatar
  • 21
2 votes
0 answers
77 views

Statistical model of a website

I know that HMMs can be used to construct statistical models of text. Thus, we can generate text according to this model, and compute the likelihood of a text sample under the model. What tools are ...
highBandWidth's user avatar
1 vote
0 answers
27 views

How does the order of implementation of Generative Data Augmentation and Generative Audio Super-Resoln matter? Which one should be implemented first?

I have a low quality audio dataset that I will use for classification. My goal is both to increase the quality of this dataset to make it easier to label it with supervised methods (super resolution) ...
Yalçın Cenik's user avatar
1 vote
0 answers
33 views

Expectation of energy-based model

An energy-based model parametrized by $\theta$ is defined as $$ p(x; \theta) \propto \exp(-f(x; \theta)) $$ For my specific case, it is that $f(x; A, y) = - \langle x, Ax \rangle - \langle x, y \...
Vũ Lê Thế Anh's user avatar
1 vote
2 answers
211 views

Autoregressive models for modeling conditional probabilities

I'm reading Tomczak's Deep Generative Modeling. When the author discusses auto-regressive models, he mentions that we model the probability distribution $p(\mathbf{x})$ of the data $\mathbf{x}$ as $$p(...
An Ignorant Wanderer's user avatar
1 vote
0 answers
51 views

How to implement simple VAE with sparse tensor in Tensorflow

thank you for reading. I have been attempting to train a simple VAE on very sparse 2D and 3D data. So far I have been training using dense tensors which - I think - is resulting in horrible training ...
Zephrom's user avatar
  • 11
1 vote
0 answers
114 views

Can you make a simple 2D ring with a GAN?

I am trying to model simple 2d continuous distributions with GANs. Here, I focus on a 2d distribution following a ring structure. The architecture of my networks are: ...
Florian Lalande's user avatar
1 vote
0 answers
348 views

Classical VAE not learning 2D gaussian mixture distribution using MSE loss

I've been exploring VAE for non-image data. I consider small to medium-sized continuous vector spaces and I want to learn the distribution of a dataset in that space. As a warm up exercise, I tried ...
Wilmerton's user avatar
  • 113
1 vote
1 answer
368 views

GANs - why does the generator want to minimize the loss (intuition)

I am a little bit puzzled about the following. In a generative adversarial network, we consider a binary classification problem with a binary cross-entropy loss. Now, the generator wants to minimize ...
heroxav's user avatar
  • 111
1 vote
0 answers
189 views

How to estimate joint probability or conditional probability using marginal ones?

I have 2 datasets: The 1st one gives us the probability that $m$ events occur on $n$ observations ($m$ columns for $n$ rows) The 2nd one tells us if the event occurred (1 if occurred, 0 else ; always ...
P. Steffen's user avatar
1 vote
0 answers
196 views

Synthetic data generation - GANs vs Simulator?

For synthetic data generation, does the GAN perform better than a simulator? If so, what are the limitations of the simulator? If we consider Conditional GANs, we could generate data based on the ...
EngGu's user avatar
  • 111
1 vote
0 answers
49 views

What should I expect if I train a Variational Autoencoder (VAE) with a dataset composed of identical images?

(leaving aside how pointless this might be) Am I right in thinking that, in theory, if I train a VAE with only one image (passing it over and over), the VAE should learn to recreate that image (or a ...
RR_28023's user avatar
1 vote
0 answers
80 views

Normalizing Flows notations and change of variable formula

Maybe it is a silly question, but I am wondering why in some papers the function $f$ is the mapping from $x$ to $z$ and in some others it is the other way around. Typically in this review we go from $...
glouis's user avatar
  • 227
1 vote
1 answer
1k views

When is my Wasserstein GAN-GP overfitting?

I have a hard time interpreting the WGAN-GP losses attached. At which epoch is D and/or G overfitting? The quality improves a lot overtime, yet the generator loss at later epochs does not appear on ...
Science_Cattie's user avatar