I have a large number of data sets. Each data set has something 200K data points lying in a square times a circle. The square is solid $I\times I$. The circle $S^1$ is hollow (dim 1). By reasoning from the experimental setup that produces the data, I am convinced that this collection of points is a sample from mixed noise plus gaussian, and the gaussian is confined to a small region. (We can assume this at first, but more realistic reasoning would give a mixture of a small number of gaussians, with centres pretty close together. I'll treat it as a single gaussian for the moment.) The problem is that the noise is far from uniform. Moreover, the noise distribution definitely differs from one dataset to the next.

Given one of my datasets D with k points (k varies with D) and underlying noise distribution $Noise(D)$, I have a way of rapidly producing a sample of k points drawn from $Noise(D)$. In other words, the information that comes with D is sufficient to identify $Noise(D)$ in the sense that it is possible to generate a sample of k points from $Noise(D)$.

My guess is that there are something like 100 points explained by the Gaussian, though this could be optimistic.

I'm not a statistician. How could I now estimate my unknown gaussian? It seems that this should be possible, since I "know" $Noise(D)$. Could someone explain to me how I could do something like the EM algorithm in my current situation. Also, could someone please recommend an online explanation of how EM is usually carried out?

  • +1 because it sounds interesting. ("... The square is solid $I\times I$. The circle $S^1$ is hollow (dim 1) ... I 'm not a statistician" .. I would have never guessed! :D ) More seriously, what you mean by mixed noise plus gaussian? Regarding the EM part of you question please see the following thread on Numerical example to understand Expectation-Maximization, the references cited will give you good mileage. – usεr11852 Jul 26 '16 at 0:33
  • I was making assumptions about my distributions D that were not justified. Sorry--downvote perhaps!! Now I smoothe D obtaining a distribution Av (local average of D). Then I bin the results, finding bins where D's bin>>Av's bin. This gives a comparatively small set of points that are to be looked at more carefully to detect Gaussians. Since there is a great deal of data associated with each point, one can check that the results are reasonable (for example, see if one's answers are "correct" by comparing two images visually---not an attractive "solution".) – David Epstein Aug 4 '16 at 11:06

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.