I'm interested in modeling a generative process that encourages data to be "evenly distributed" over its support, i.e. clumping of data points is penalized.

For example, if I have a mixture distribution and I set all components to have equal weight, it will tend to generate points equally between all the components, but this property isn't enforced. The problem here is that if we evaluate a data set under this likelihood, it gives the same probability to a data set that is evenly distributed as it would to a data set that is clumped under a single component.

To illustrate a "better" model that does penalize clumping, consider the following generative process. Beginning with a mixture distribution with $N$ components, sample a point in the standard way: first sample an index $i$ from the component weights, and then sample a point from that component. Now decrease the weight of the $i$-th component to be a certain percentage of its original weight (call this percentage $p$), and distribute that mass to the other $N-1$ components. Repeat $M$ times.

This process generates points from the mixture distribution in a way that encourages points to be evenly distributed between the components. The parameter $p$ adjusts how strongly this even-distribution of points is enforced--when $p=1$, this is a typical mixture distribution. When $p=0$ and $M = N$, this enforces a strict one-to-one correspondence between data points and mixture components. The problem with this process is that although it's straightforward to sample from, evaluating the data likelihood under this model appears to be intractable, due to the need to consider all possible orders in which the data could have been selected.

Are there any well-known models that penalize clumping in this way?

  • 1
    $\begingroup$ It seems to me that picking points uniformly and independently at random does exactly what you're looking for. It sure has a simple likelihood function :-). $\endgroup$ – whuber Sep 14 '11 at 21:08
  • 1
    $\begingroup$ The important quality of process I described is precisely that it doesn't generate points independently. It uses the locations of previously generated points to avoid generating new points in the same location. I'm guessing the lack of independence in the samples would make this model too problematic to be used in practice. $\endgroup$ – redmoskito Sep 15 '11 at 16:47
  • 1
    $\begingroup$ You might look into the GSRT method: oregonstate.edu/dept/statistics/epa_program/docs/…, oregonstate.edu/dept/statistics/epa_program/docs/… $\endgroup$ – whuber Sep 15 '11 at 16:58
  • $\begingroup$ The central limit theorem will ensure (with high probability) that this isn't a problem for the iid process with large M. If M is small enough, you can always directly enumerate sample paths. Then again, if the iid mixture process is being used as a prior for a latent Bayesian method, the data-generating function ("likelihood") would, presumably, encourage even distribution if the data were to warrant it, even in the case of a small sample. At the worst case, I think you could represent this process as an HMM and use the standard techniques. $\endgroup$ – Timothy Teräväinen Apr 26 '17 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.