1
$\begingroup$

I am looking for a way to identify possible distributions within a set of data points.

  • assuming I have a 1-dim mixed vector of points $d_j$ within $[0; 300]$
  • the points can be from different distributions for example:
    • log-normal distribution with $LN (10, 0.5)$
    • exponential distribution with $\lambda = 0.6$
    • other distributions like Erlang (or Gaussian) [spec. it is about durations of failures]

My aim is to tell if the data points are from one distribution or maybe from two different. But so far I faced some problems for which I am looking for help:

  • if the distributions overlap, some common clustering algorithms fall short, here I tried using mode detection (LPMode) but I am not sure if there might be a better way
  • can I take advantage of the fact that I can restrict the choice of distributions to choose from?
  • I had a look into AutoClass and was wondering if there are some advances in the Bayesian methods I should have a look into?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The EM principle can be applied to other distributions as well, not only with Gaussian Mixture models.

It will give you a goodness of fit (how well the data is explained), so this may serve as a starting point for evaluating the match. You probably want to also require each distribution to account for, e.g., 10% of the data at least, to avoid false results that just capture some small 'bump' rather than a separate distribution.

It may be worth exploring robust estimators instead of MLE if you assume the data to be dirty.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Ok but now I have two more questions: 2) will this even work on 1-dim data, because every example so far mentions multi-dim. And can this be used to identify different distributions? Maybe I am misunderstanding sth but I have only encountered EM to look for multiple distributions of the same type; not a mixture of different distributions EDIT: found an example using a mixture of gaussian and exponential - so this is clear $\endgroup$
    – balleny
    Commented Nov 28, 2017 at 9:43
  • $\begingroup$ Yes, it will work on 1-dimensional data. Just use a univariate distribution instead of a multivariate distribution. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Nov 28, 2017 at 16:12
  • $\begingroup$ But it would require me to know the type of distributions involved a priori or test everything and do a goodness-of-fit evaluation, wouldn't it? $\endgroup$
    – balleny
    Commented Nov 29, 2017 at 13:00
  • $\begingroup$ Yes. It's one dimensional, meaning it's fairly manageable. Try different candidates, keep the best. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Nov 29, 2017 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.