3
$\begingroup$

I have near about 50 files(each file corresponds to a patient) with 4 columns -

chromosome     start.position       stop.pos       value

First 3 columns in all 50 files are same and fourth column is value which is different (or can be same for some) for all patients. Basically this value corresponds to copy number (structural variation in genome). I want to apply some test (or if you can suggest any other procedure) to find out values which are deviating as compared to others. Example-

file1
chromosome     start.position       stop.pos           value
      1              10                  110             4
      2              100                 200             5
      2              500                 600             0.5

file2

chromosome     start.position       stop.pos           value
      1              10                  110             2.5
      2              100                 200             6
      2              500                 600             0.6

file3

chromosome     start.position       stop.pos           value
      1              10                  110             3
      2              100                 200             5.5
      2              500                 600             3.5

file4

chromosome     start.position       stop.pos           value
      1              10                  110             2
      2              100                 200             0.9
      2              500                 600             3

file5

chromosome     start.position       stop.pos           value
      1              10                  110             8
      2              100                 200             4.5
      2              500                 600             2.5

So in the output, file5 has significant value (as compared to same position of other files) at row1, file4 has significant value at row2 and file1 and file2 has significant value at row3.

I have 1 solution in my mind- pick a row from each file and take the average of values. Then compare this average with each file and return the position if it is significant as compare to average. But I feel there should be some better solution for this. Can you please suggest any test or any other procedure that you will use to complete this task.

Thanks in advance.

EDIT: After reading Joel's comment.

Yes I have a specific question in my mind. As I mentioned these files corresponds to each patient, so I am trying to find if a patient has significantly increase(or decrease) in value at certain position (i:e row). Ok we can talk about only 1 row now. Just consider 1 position (1 row), take the values ( in this case, for first row we will have 5 values as we have five files) and from these values check which value is significantly deviating from others. So if you can suggest me solution for 1 row, I can do it for all the rows one by one. Please let me know if it is not clear.

Improve this question
$\endgroup$
  • $\begingroup$ Your research question is not clear to me nor the theoretical interrelationships in the data. If you were to block on a variable (i.e., do a repeated measures analysis) would you block on rows or patients or both? Would you like to do one test to see if there is any statistical difference(s) in the data or do you have specific questions in mind? Are you interested only in central tendency or is dispersion of interest to you also? With a little more detail, perhaps someone here can be of help. $\endgroup$ – Joel W. May 7 '12 at 9:51
  • $\begingroup$ i would have a look here: stats.stackexchange.com/questions/1519/… $\endgroup$ – user603 May 11 '12 at 8:21
2
$\begingroup$

If you are looking for an outlier in a small test of variables there is Dixon's ratio test and Grubbs' test that are designed to detect outliers in otherwise normally distributed samples. Dixon's test is simpler and has often been applied to find outliers in small groups. I published a paper on Dixon's test to show that it works well for some non-normal populations as well in small samples (3 to 5). There are variants of the test that handle situations wher you might have two outliers and the second largest can mask the largest value. My paper "A Note on the Robustness of Dixon's Ratio Test in Small Samples" The American Statistician 1982.

Improve this answer
$\endgroup$
  • $\begingroup$ Bingo! Thanks a lot. This is something I was looking for. Just one question, Does Dixon test can detect more than 1 outlier? $\endgroup$ – Vikas May 7 '12 at 12:11
  • $\begingroup$ I should add that if you apply the test repeatedly to more than one row you have a multiple testing issue to worry about. The test works like this: You have a sample X1,X2,...,Xn with order statistics X(1),.....,X(n). To test for the largest observation being an outlier you take the ratio [X(n-1)-X(1)]/[X(n)-X(1)]. The test uses the null distribution for this ratio when the observations all come from a single normal distribution. If the largest observation is a single outlier then the spacing between X(n) and X(n-1) will be larger than the expected spacing for a normal distribution. $\endgroup$ – Michael R. Chernick May 7 '12 at 13:54
  • $\begingroup$ is it [X(n)-X(n-1)]/[X(n)-X(1)]? $\endgroup$ – Vikas May 7 '12 at 14:01
  • $\begingroup$ Therefore the ratio will be relatively small. But if you have two large outliers X(n-1) masks or hides X(n) and the test will probably fail to detect any outliers. Dixon has other ratio tests to handle this problem. For example the ratio [X(n-2)-X(1)]/[X(n)-X(1)] will take care of that problem and indicate that you have two outliers. If the outliers are low values you just take [X(n)-X(2)]/[X(n)-X(1)] and the distribution theory is the same. Dixon's paper came out in 1950 so it has a long history. $\endgroup$ – Michael R. Chernick May 7 '12 at 14:03
  • $\begingroup$ The problem may be to pick the "right" ratio test which would mean you have an apriori notion of how many outliers you have. $\endgroup$ – Michael R. Chernick May 7 '12 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.