# Statistical test suitable for small dataset (n=25)

I have a dataset of cases and age & sex matched controls (n=25). I need to do a statistical test to find the significantly different incidences of ICD-9 codes using a p-value or similar measure.

What test should I use in this scenario and how should I define significance ? Given the low number of samples, should I use a p-value higher than 0.05 ? R script/pro-tips to deal with small data sets are appreciated .

This is my tab-delimited data file looks like.

ICD-Code Description Cases Controls p-value
401.1 Hypertension, benign 14 3     ?
405 Secondary hypertension 20 1     ?
250 Diabetes mellitus      20 20    ?
.
.
.


EDIT: I am trying to understand the trade-offs in selecting an appropriate p-value to define the significance with my dtaand a suitable multiple comparison method here. Any suggestions on such aspects ?

• Perhaps a chi square test for each ICD-9 code and correcting for multiple comparisons in some way? chisq.test() in R. – Matt Albrecht Nov 10 '11 at 2:30
• With matched case-controls, you may need more data than this data file. With matching--taking 1:1 matching for example--information comes from where the case and the control are discordant. For instance one has hypertension and the other doesn't, or vice versa. – Ming-Chih Kao Nov 10 '11 at 2:47
• Thanks Matt. Chi-square test is fine, but I am trying to understand the trade-offs in selecting an appropriate p-value to define the significance and a suitable multiple comparison method here. Any suggestions on such aspects ? – Khader Shameer Nov 10 '11 at 2:47
• Do you only have this kind of "aggregate" information? If you had information about the pairs you could set up a table (cases vs controls) for each ICD code and then take advantage of the pairing by using McNemar's test. This and using an exact version of the test could help dealing with the small sample size. – psj Nov 10 '11 at 14:41
• @Khader, the pairs are defined by the matching. For each case there should be one control with same sex and age. The table format for this setup could be (for hyptertension, htn): row1, col1: number of pairs having case and control no htn; row1, col2: number of pairs having case no htn, control with htn; row2, col1: number of pairs having case with htn control no htn and finally row2, col2; number of pairs having case and control with htn. – psj Nov 10 '11 at 15:29

I've given an example below in R, based on what I think your data is, that's built on random sampling and then applying a chi-square test. Then, with the simulation in mind, you can think about hypothesis testing based on the other answers. Although, I understand that sometimes reviewers will want to see tests of significance for all the demographic variables. However, if you had a specific hypothesis in mind then you should test that and maybe leave the others alone or correct for multiple comparisons. For example, if based on your reading of the literature, you thought it reasonable that a particular, or set of particular, ICD-10 codes would be different across your groups, then just test those ones. If you're just looking to see, have a look at multiple comparison procedures like the Bonferroni, Holm, false discovery rate etc.

Example

df1<-data.frame(icd.code=paste("icd", 1:20, sep="."),
cases=sample(1:25, 20, replace=T),controls=sample(1:25, 20, replace=T),
total.cases=25, total.controls=25)

chis.p<-sapply(df1$icd.code, function(x) { int1<-with(subset(df1, icd.code==x), rbind(c(controls, total.controls-controls), c(cases, total.cases-cases))) int2<-chisq.test(int1) int3<-c(int2$statistic, int2$p.value) }) chis.p<-t(chis.p) df1$x.square<-round(chis.p[,1], 2)
df1\$p.val<-round(chis.p[,2], 3)

• Thanks a lot Matt. My hypothesis is that a set of ICD-10 codes would be significant in cases when compared to the controls. – Khader Shameer Nov 12 '11 at 3:51

You may wish to perform a power analysis or at least think informally about the relative importance of Type I and Type II errors in your set-up. That would help you decide on an appropriate p-value.

It seems to me that you do not have a hypothesis in mind. In that case, why do a hypothesis test in which you have to claim significant/not significant and (arguably) have to control for the inflation of type I errors in the multiple tests? Worrying about a critical cutoff for significance is only a partial solution.

Instead, perform significance testing and rank the interesting effects on the basis of the size of the p values. Don't worry about significant/not significant. Look at the results and form hypotheses to test with new data.

• I like your 1st paragraph. For the second, how about "Instead, assess the magnitude of the differences and rank the interesting effects on the basis of these magnitudes." Which could be done using standardized residuals from chi-square tests, as one way. Leaving p out of it seems most consistent with your 1st paragraph. – rolando2 Nov 11 '11 at 10:56
• Thanks Michael. I do have a hypothesis. My hypothesis is that a set of ICD-10 codes would be significant in cases when compared to the controls. Can you give an example of the significance testing ? – Khader Shameer Nov 12 '11 at 3:54
• I agree with rolando2 that the rank of difference is useful. I was thinking that the p values would give an index of the magnitude of the differences scaled for their 'convincingness'. It would be interesting to see if there was any important differences in the ranks assigned by p values from, say, t-tests and the standardised residuals suggested by rolando2. – Michael Lew Nov 13 '11 at 9:25