# How to take random draws of a low-entropy “meta-random” distribution

I'm working on a simulation study, and for it I'd like to be able to generate random draws from a random multivariate distribution. I'm looking for something pretty chaotic, in the sense that it's got multiple states with low within-state "entropy" (in a hand-waivey sense of the term).

I'm not entirely sure how to pose the question, so perhaps let me start with a few constraints:

• It can be defined in real numbers, with arbitrarily many dimensions
• These dimensions vary from being almost-independent to dependent
• The pattern of dependence between any two dimensions varies from global to highly local. In other words, $x_1$ and $x_2$ may be highly correlated along some of their joint support, but uncorrelated along other parts of it. The change in correlation over the support need not be constant, or even monotonic.
• Likewise, the pattern of dependence need not be restricted to the first moment. For example, the kurtosis of $x_2$ might depend on the variance of $x_4$.

I've got some vague ideas about how such a thing might be approached, but they're not really all that well-developed. For example, I could draw random numbers for each variable, and calculate the entropy of the distribution relative to a maximum-entropy distribution with 5+ moments. I'd then retain those distributions with low entropy. I'm not sure how feasible this would be however, or even if it'd give me what I want.

Is there anything already in this space? If there were, it seems like it'd be useful to people working with nonparametric statistics and machine learning.

How about Gaussian mixture models (GMMs)? These have the advantage of already being in common use. They seem to satisfy the requirements you listed (see below). They can also be used to approximate any other distribution as the number of mixture components grows.

I'd like to be able to generate random draws from a random multivariate distribution.

You can easily generate a random number of components, components weights, means, and covariance matrices. For generating random covariance matrices, it's convenient to parametrize them by the eigenvectors and eigenvalues. Generate random eigenvectors by orthonormalizing a matrix containing random elements drawn i.i.d. from a standard normal distribution. Or, specify them directly if you want control over the direction of the principal axes. Generate eigenvalues according to some function. For example, an exponentially decaying function, where the decay constant controls how elliptical the resulting multivariate Gaussian will be. Construct the covariance matrix $C$ using the eigenvectors (columns of matrix $V$) and eigenvalues (stored on the diagonal of matrix $\Lambda$): $C = V \Lambda V^T$. An alternative would be to treat the GMM more like a kernel density estimator; use many spherical Gaussian components and choose their locations such that they sum up to some desired shape.

It can be defined in real numbers, with arbitrarily many dimensions

Check.

These dimensions vary from being almost-independent to dependent

Check. For example, consider a single Gaussian component. A diagonal covariance matrix will give you independent variables. A highly elliptical, non-axis-parallel Gaussian will give dependent variables. Of course, you can get much more complex patterns using multiple mixture components.

The pattern of dependence between any two dimensions varies from global to highly local. In other words, $x_1$ and $x_2$ may be highly correlated along some of their joint support, but uncorrelated along other parts of it. The change in correlation over the support need not be constant, or even monotonic.

Check. For example, imagine a 2d Gaussian mixture model with two components. One component has a highly elliptical shape that's oriented at an angle w.r.t. the axes. The variables will be 'locally dependent' in this region. The second component is located adjacent to the first, and has a diagonal covariance matrix. The variables will be 'locally independent' in this region. As before, much more complicated patterns are possible using more mixture components.

the pattern of dependence need not be restricted to the first moment. For example, the kurtosis of $x_2$ might depend on the variance of $x_4$.

This should be possible because a GMM can approximate any other distribution. I don't know how to explicitly choose a set of parameters that would implement a given relationship between kurtosis along one dimension and variance along another, but it seems possible in principle.

If I recall correctly, there isn't a closed form solution for the Shannon entropy of a GMM, although algorithms exist for iteratively computing/approximating it. This would let you avoid having to estimate entropy by sampling points (which is difficult in high dimensions). I vaguely recall having seen a paper where they proposed using Renyi entropy as an alternative for GMMs, for which they gave a closed form solution.

You might want to place some kind of constraints on the parameters. For example, you could constrain means/covariance matrices such that the distribution has most of its mass concentrated in a single, compact region instead of in isolated clusters. From the general description you gave, it's hard to say what specific constrains you'd want to implement. But, in general, one approach is rejection sampling (as you described): pick some criterion, draw the parameters of the distribution, keep the distribution if it satisfies some criterion of interest (e.g. about entropy), otherwise discard and repeat. In some cases, it might be possible to draw parameters in such a way that all resulting distributions satisfy your criterion (or at least give a higher fraction of accepted draws).