Timeline for Detect outliers in mixture of Gaussians
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 12, 2012 at 23:04 | comment | added | Peter Flom | Two points about that Wikipedia example: 1) It's only for a mixture of 2 Gaussians, you specified it could be more. 2) It would NOT help with your problem, I think, because it would have to include any supposed outliers in the data, and thus estimate mixtures of Gaussians that include the outlier. | |
| Feb 12, 2012 at 21:05 | comment | added | whuber♦ | I think the key to resolving this is the implicit criterion that each mixture component has enough probability to guarantee that it contributes a large number (10 or more, e.g.) of the data. Then isolated clusters (of one to nine points) identified by fitting a mixture model would violate this assumption and constitute either "outliers" (for extreme values) or "inliers" (for non-extreme values). | |
| Feb 12, 2012 at 21:02 | comment | added | D.W. | Wikipedia has a brief example of how EM can reconstruct a mixture model, after being told only that it is a mixture of Gaussians. This tutorial has more technical details. | |
| Feb 12, 2012 at 12:48 | comment | added | Peter Flom | I don't see how EM could reconstruct the mixture model, if told only that it is a combination of normal distributions. Maybe it could, but I don't see how it could do so precisely. And something that is an outlier from one mixture of normals would not be an outlier from another mixture of normals. | |
| Feb 12, 2012 at 0:21 | comment | added | D.W. | I don't understand why you think your example proves the problem is not solvable. In your example, presumably fancy methods like EM could reconstruct the parameters of the mixture model (100 observations from the $\mathcal{N}(50,100)$ distribution should be plenty), compute $p$-values for each observed value, and then identify outliers. (In your example, once we know the parameters, 317 is only 2.7 standard deviations above the mean 50, so not an outlier.) So it seems it should be possible to detect outliers. I don't follow why you've concluded it is impossible. | |
| Feb 11, 2012 at 21:58 | comment | added | Peter Flom | I don't think the additional restriction really solves things enough to make the problem solvable in an automated way. Look at my example. x <- c(rnorm(9900,10,10), rnorm(100,50,100)) quantile(x, .999) x[x>175] The first time I tried this, I got a maximum of 317.8, and a 2nd highest of 233, then things were tightly bunched. Is 317 "far" from 233? I think so. But it's a combination of 2 normals | |
| Feb 11, 2012 at 17:46 | comment | added | D.W. | Thanks. Good points. See my update to the question for more information that explains how you can have outliers. Automated searching: yes, I realize that automated search is only a first step. A human will examine all items that have been flagged as a possible outlier, and there are other ways (out of scope for this question) for identifying outliers. However, I don't want to bother the human too much more than necessary. | |
| Feb 11, 2012 at 13:17 | history | answered | Peter Flom | CC BY-SA 3.0 |