Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite
3

I am a bit new to this field. So I needed help in finding out which topic should I focus on for achieving this.

Suppose I have N dependent random variables. I have n samples of each of these random variables. Now I want to check if the kth sample follows the pattern that is suggested by the previous (k-1) samples.

For example if there are 2 variables - X1 and X2. For the first 10 samples lets say X1 > X2. But if for the next sample X1 < X2, then this sample is anomalous. I need to detect such an anomalous sample.

Can you tell me what statistical machine learning concepts should I study in order to understand which techniques are applicable in such a scenario?

From my limited knowledge this appears to be a classification problem that must be attacked with unsupervised technique.

share|improve this question
1  
I find the question too vague. You need to define "anomalous", e.g. as you did in the example. In statistics terms, you need to define a model for "normality" (i.e. lack of abnormality)... – Xi'an Nov 9 '11 at 15:56

2 Answers

up vote 5 down vote accepted

This sounds like a job for the Mahalanobis Distance. You would apply this by estimating the population covariance and mean vector using the previous $(k-1)$ samples (assuming $k > N$, where $N$ is the dimension of your observed vectors), and then computing the Mahalanobis Distance. A google search reveals a lot of hits on this topic, Bartkowiak seems like a decent starting place.

share|improve this answer
I tried calculating with a contrived dataset having 100 samples drawn from a hypergeometric distribution. The dimension was 6. I get the following error in R: > mahalanobis(x=c(28, 22, 3, 20, 4, 2), mean1, cov1) Error in solve.default(cov, ...) : system is computationally singular: reciprocal condition number = 2.16688e-018 – iamrohitbanga Oct 25 '11 at 2:29
It works for me: mean1 <- c(0,0,2,3,4,1) cov1 <- cov(matrix(rnorm(200*6),ncol=6)) mahalanobis(c(28,22,3,20,4,2),center=mean1,cov=cov1) Maybe your covariance is singular? check with rcond(cov1) (should give a reasonably small number) – shabbychef Oct 25 '11 at 4:37
yes it does give a small number. – iamrohitbanga Oct 25 '11 at 9:31
mat1 = t(rMWNCHypergeo(100, m=c(10000000, 10000000, 10000000, 10000000, 10000000, 10000000), n=10000, odds=c(30, 20, 10, 30, 5, 5))); cov1 = cov(mat1); mean1 = c(mean(mat1[,1]), mean(mat1[,2]), mean(mat1[,3]), mean(mat1[,4]), mean(mat1[,5]), mean(mat1[,6])); I used Hypergeometric distribution. What should we do when the cov matrix is singular? – iamrohitbanga Oct 25 '11 at 17:03
2  
I get a very small number when I look at the rcond(cov(mat1)) using your example mat1, like 1e-17. Your covariance is rank 5, so any techniques which invert it are going to fail. Looking at rowSums(mat1) I see the problem: the columns of mat1 are not linearly independent--they all sum to 10000. This is probably due to how your Hypergeometric generation process works (i.e. each ball drawn has to be from some class). To get around this problem, subselect the first five columns of mat1 and ignore the last one (it is completely determined by the first five). hth. – shabbychef Oct 25 '11 at 17:50
show 2 more comments
up vote 4 down vote
+25

This sounds to me more like a change point problem. You should investigate the bcp package, http://cran.r-project.org/web/packages/bcp/index.html, as well as strucchange, http://cran.r-project.org/web/packages/strucchange/index.html

John Emerson maintains the bcp package, and when I met him, he seemed very willing to engage the community and potential users: john.emerson at yale.edu

Alternatively, if your data are time series data, you may wish to investigate the qcc package, and read some of Douglas Montgomery's work on Statistical Quality Control.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.