I have a set of probability distributions. I am also going to be sampling some data and I would like to determine which distribution the data fits to best. But there are some caveats:

  • The probability distributions are beliefs moreso than probability distributions.
  • I will have very small samples. This means that I would like to fit the sample to the distribution (or belief) after a minimal number of samples. I would love for it to happen in real time i.e. to fit the sample to the distribution after 1 sample, then 2 samples, then 3 samples etc. and then at some point to say that this sample is definitely from this distribution/belief
  • The distributions/beliefs are discrete. In fact, I would describe the distributions/beliefs as being of the family of the multinomial distribution
  • There is also the probability that the distribution/belief which the sample is best suited to may change after N samples, where N is random. So as an example, lets assume that the data is best suited to distribution/belief number 1, and that there are N samples: at first we are uncertain which distribution the samples falling into, but then after drawing a few (say, k samples, where k

Given the above, I am considering a few options for how to do this:

  • Maximum likelihood expectations seem like a natural way forward
  • I also hear that the EM algorithm is great for problems of this kind i.e. when the probability distributions are uncertain and can best be described as a belief

My questions are then:

  • What would be the best way to fit the data to one of the distributions/beliefs? Is the EM algorithm indeed better for problems of this nature? Or is there a better way?
  • List item
  • $\begingroup$ "at some point to say that this sample is definitely from this distribution/belief"... this isn't possible. In some very restricted, relatively artificial situations (e.g. taking a Bayesian viewpoint but with a limited set of possible models) we may with sufficiently large samples be able to say that one choice is much more probable than the others, but in the more typical situation of just trying to identify some distribution, we simply can't do this. Effectively rule some distributions out, sure, but to say it's this one and no other, no. ... ctd $\endgroup$ – Glen_b Aug 26 '17 at 2:31
  • $\begingroup$ ctd... in particular, if you're looking to say it's this multinomial and not any other multinomial, you'll have no way to exclude an infinite number of multinomials that would be sufficiently near any specific one that fits well. If you take a Bayesian viewpoint and you have a list ... multinomial A or multinomial B or multinomial C and nothing else is possible (which I again say is artificial), then if the data work out that way you can get some way toward choosing exactly one as much more probable (but never definite). $\endgroup$ – Glen_b Aug 26 '17 at 2:34
  • $\begingroup$ @Glen_b thanks this is very true when i think about it. then i guess my question is - given a small sample size, what is the best technique for maximizing the likelihood of the values of the parameters of the distribution type which you know that the sample fits into? Furthermore, given this technique, how small a sample size is enough for one to get a statistically significant assurance of fit/suitability? $\endgroup$ – user5211911 Aug 26 '17 at 13:06
  • $\begingroup$ @Glen_b ctd...answering my own question, I believe that the em algorithm may be suitable for this sort of work. if i'm correct, what sample size does the em need for convergence, for different numbers of "clusters"? I believe this may be a tough one to explain and that there will be lots of theoretical conditions for convergence, therefore if you could synthesize or possibly direct me to relevant papers/websites/readings/videos? Thanks, really appreciate it $\endgroup$ – user5211911 Aug 26 '17 at 13:12
  • 1
    $\begingroup$ EM is a good choice for estimating parameters in mixture models. However, your questions are not the sort of thing that can be answered briefly so not really suited to a question in comments; outside the original papers on EM I don't really have specific references to suggest (others might). $\endgroup$ – Glen_b Aug 27 '17 at 2:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.