3
$\begingroup$

A colleague has developed a treatment for to "prevent falls" in cognitively impaired, psychiatric patients. Since this would be very useful treatment in this population, we especially do not want to make a Type II error (i.e., fail to reject the null, when we should reject it).

Since the data is not normally distributed, another colleague evaluated the full data using (appropriately, I believe) the Wilcoxon test, and did not find significance. There may be valid methodological reasons for this, which I may follow-up on later with another question.

I was concerned about committing a Type II error, and obtained some preliminary data, which I have below. These data reflect pre/post scores (# of "falls") for the same patients (no control group), so should be considered "paired" and not independent:

pre <- c(9,8,37,12,8,3,4,4,3,5,4,8,4,8,9,11,2,4,0,0,5,12,10,2,8,3,0,22,1,0,0,5,0,3,1,5)

post <- c(10,8,6,4,5,2,4,4,2,2,1,7,2,1,3,9,2,2,0,0,6,16,4,3,4,7,0,10,3,0,0,4,0,1,1,5)

When I ran a bootstrapping procedure on this preliminary data (adapted from Crawley, The R Book, p. 385)

preBoot <- numeric(10000)
for (i in 1:10000) {preBoot[i] <- mean(sample(pre, replace=T)) }
quantile(preBoot, c(0.025, 0.975))

and compared the post mean to the bootstrap estimate of the sampling distribution of the mean, I found that the treatment did have a significant beneficial effect. To evaluate significance, I simply took the quantiles for the sample estimate at 0.025 and 0.975; is this correct or am I confusing what I would do with a normal distribution with the distribution of the sample estimates of the mean?

Also, using wilcox.test in R on the preliminary data (i.e., 

wilcox.test(pre, post, paired=T, exact=F)

shows this to be significant.

I would like to know, before I go further, did I use the bootstrapping procedure correctly, and is this a legitimate test for these type of data?

Are there other tests we should consider, and what would be the best way to report this? I am especially interested in methods that would allow us to obtain confidence intervals.

Also, I see in this previous question Wilcoxon one-tailed test the response indicated that "keep in mind that it's generally not advisable to use one-tailed tests," but if I'm interested specifically in fewer falls after the treatment intervention, wouldn't a one-tailed test be appropriate?

Additional Info Update: I just found a wonderful review on count data and analysis by Neal Alexander, Review: analysis of parasite and other skewed counts , accessed via PubMed http://www.ncbi.nlm.nih.gov/pubmed/22943299 which discusses the issues I've been facing in a very accessible manner (and it is free online). Other reading this question may also find this quite helpful.

I'm still digesting this info. This probably belongs in a new question, but in essence, I believe in my field (clinical psychology) the standard way of looking at this data would be with a Wilcoxon test, with perhaps a square root transformation and a t-test running in second place. Most people currently don't use R, and so don't seem to be aware of or use bootstrapping, which I actually believe would be better than the above two methods. If anybody has further info/or information to the contrary, I would appreciate it).

Improve this question
$\endgroup$
9
  • $\begingroup$ Your data are counts. This would suggest you use a form of model that's directly relevant to counts. $\endgroup$
    – Glen_b
    Mar 3, 2014 at 4:49
  • $\begingroup$ @Glen_b - Yes, these data are counts (i.e., # of "falls") in this group pre- and post-intervention. This would mean that the Median Sign test would not be appropriate, and that possibly (help me out here) the Wilcoxon signed rank test would also not be appropriate? $\endgroup$
    – Woodstock
    Mar 3, 2014 at 15:13
  • $\begingroup$ Both tests assume continuous distributions; you're going to lose some of the nominal properties (and give up some power) when using them with count data. For example, you won't have the significance level you think you have. $\endgroup$
    – Glen_b
    Mar 4, 2014 at 6:37
  • $\begingroup$ @Glen-b re: count data, I might have misunderstood your original point. We aren't interested in counts of "y/n" we are specifically looking at did the intervention result in fewer #'s of falls (these "falls" are typically not experienced by 70% of our population. These people were specifically selected for their high number of falls over a long period of time, which appear to be refractory to any other treatment question). I'm conversing with my colleague, to see if we can get the time periods over which these "falls" occured, and thus look at comparing incidence rates pre- and post-. $\endgroup$
    – Woodstock
    Mar 4, 2014 at 14:37
  • $\begingroup$ Yes, understood. The issues I raised apply to that. $\endgroup$
    – Glen_b
    Mar 4, 2014 at 14:45

1 Answer 1

Reset to default
5
$\begingroup$
  • You investigate a paired data situation, as you mentioned, however, you treated it like independent. You should run the bootstrap on differences of pre/post measurements of each patient. Then, see, whether the interval contains zero.
  • Although it's not generally advisable, in the situation as you describe it applying the one-tailed spcification is reasonable.
  • Wilcoxon Signed Rank test is correct method, as is Median Sign test. Moreover, you can consider transforming your count data by taking square roots and then performing t-test.
Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ :re: your point #1, "You investigate a paired data solution," -- thank you for pointing that out. Using the bootstrap method, I should be taking the difference of pre- & post- scores, running the bootstrap, and then seeing if the 95% CI contains 0. Is that correct? $\endgroup$
    – Woodstock
    Mar 2, 2014 at 17:43
  • $\begingroup$ Also, I had looked at using the square root transformation and then the t-test. I can't use a log transform, because of the 0's in the data (which may get resolved when we iron out a methodological/study issue). Thanks for suggesting the Median Sign test. $\endgroup$
    – Woodstock
    Mar 2, 2014 at 17:54
  • $\begingroup$ Yes, you should do that. $\endgroup$
    – Germaniawerks
    Mar 2, 2014 at 17:55
  • $\begingroup$ I just accepted your answer - I can't upvote it yet, because I don't have enough reputation points. $\endgroup$
    – Woodstock
    Mar 2, 2014 at 18:22
  • $\begingroup$ Regarding first point: Is it really necessary to run bootstrapping? Why not just plot the histogram of the paired differences? $\endgroup$
    – Michael
    Nov 16, 2020 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.