4
$\begingroup$

Typically when I see generative models, e.g., Latent Dirichlet Allocation (JMLR) or Linear/Quadratic Discriminant Analysis (wikipedia LDA), they are probabilistic models that belong to the exponential family (stackexchange regularity conditions, wikipedia exp fam).

Recently, I read in Murphy's Probabilistic Machine Learning textbook (Probabilistic ML) about the principle of maximum entropy (wikipedia maxent) and the relationship to exponential families. It turns out that exponential families maximize entropy given certain constraints on the cumulants, but I am still chewing on this idea. Maybe this reason alone is why exponential families are popular in generative models?

However, if we are most interested in the covariance structure of a generative probabilistic model for certain datasets, is there a nonparametric generative model that might model certain data better than a parameterized (perhaps exponential family) generative model?

Perhaps this is a really bad question, and if I knew more about nonparametric statistics, bayesian statistics, and copulas (wikipedia copulas) the answer would be clear. Any insights into a nonparametric generative model in general, or specific to modeling covariance structure in datasets would be greatly appreciated!

Background:

This question came up while trying to think about assumptions for second order moments for measuring association: stackexchange second order moment question. If I am most interested in modeling a complex covariance structure in a dataset, and want to use a continuous (regarding CDF) generative model, then is the principle of maximum entropy strong enough to only consider a multivariate normal distribution (when we assume finite second order moments)? Or is there a nonparametric generative model that may model complex covariance structure more robustly? Are copulas considered nonparametric models?

Related stackexchange questions that do not exactly answer the question as I understand it are:

Generative Models and regression problems

Markov Random Fields and Exponential Families (This seems very relevant to me, however I still do not think it answers the question involving a nonparametric generative model. Unless I am mistaken, a mixture distribution is still a parameterized distribution.)

Bayes and Generative Models

Bayesian Posterior and Generative Models

Improve this question
$\endgroup$
5
  • $\begingroup$ Could you explain what you mean by "model better"? Arguably, parametric models are "better" than nonparametric ones in many ways, such as allowing explicit computer generation of synthetic datasets and permitting clear analysis of the power of tests (because the alternatives are relatively limited in scope). Do you equate "more robustly" with "better," perhaps? By "copulas" do you mean all copulas or, perhaps the parametric families described in books on the subject? How do you see maximum entropy as implying you only need multivariate Normal models for data? $\endgroup$
    – whuber ♦
    Commented Feb 24, 2020 at 20:22
  • $\begingroup$ Of course, I definitely did not write as precisely as I should have. By 'model better', I am really trying to say 'better capture the intrinsic covariance structure'. So perhaps measured in terms domain expertise for certain datasets, or some type of cross validation. When I think of nonparametric models, I naively think of more generalized functional forms than parameterized models. And so in my naive picture I imagine a nonparametric generative model being able to capture complex dependence structures. Of course then the problem comes into fitting the model etc. $\endgroup$
    – Sleepy 17
    Commented Feb 25, 2020 at 3:33
  • $\begingroup$ Regarding copulas, I simply do not know much about copulas, except what can be found on the wikipedia page and a few tutorials from google searches. I couldn't specify better than simple "copulas", and perhaps they aren't relevant at all. $\endgroup$
    – Sleepy 17
    Commented Feb 25, 2020 at 3:34
  • $\begingroup$ Regarding the last question: If we constrain a multivariate generative model to have a continuous CDF, a certain mean vector, and a certain covariance matrix (second cumulant). Then, as I understand it, the multivariate normal with this mean vector and covariance matrix is the distribution that maximizes entropy. Certainly if I drop the continuous CDF requirement things could change. Perhaps not, I would need to check this with some calculations. My question was meant to get at: Is this sufficient for using MVN distrib. so often, for continuous-valued data with finite second moments? $\endgroup$
    – Sleepy 17
    Commented Feb 25, 2020 at 3:41
  • $\begingroup$ One important reason for using nonparametric models is because the underlying distribution may depart from Normal (or whatever the max ent solution is) so very strongly that the parametric assumptions are just too poor or even misleading. $\endgroup$
    – whuber ♦
    Commented Feb 25, 2020 at 15:32

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.