3
$\begingroup$

I was looking to get some advice on some analysis using R, I want to ensure I am doing everything correctly.

I want to compare test scores between two groups (A and B), I have two tests but wish to look at each individually, i.e compare scores between group A and B for Test 1, then do the same for Test 2.

Here is my example data:

exampledata1 <- read.table(text="Unique ID,Test.1,Test.2,Group
544001,3,1,A
668149,0,0,A
314205,0,1,A
646843,1,1,A
599014,0,1,A
502115,1,0,A
123725,2,2,A
170541,3,3,A
181769,1,1,A
928772,0,0,A
448116,0,1,A
287093,0,0,A
754849,1,1,A
862914,0,0,A
390557,0,0,A
749690,0,0,A
423885,1,0,A
831307,0,0,A
642633,1,,A
960592,0,0,A
162882,1,1,A
762726,0,0,A
463306,0,1,A
236706,1,1,A
979328,0,1,A
590783,3,3,A
821588,1,0,A
112601,1,1,A
921085,2,1,A
336733,0,0,A
315681,0,0,A
237933,1,1,A
698807,0,0,A
720256,1,1,A
525437,0,1,A
735806,0,0,A
260575,1,2,A
763688,1,1,A
114882,0,1,A
525912,3,1,A
972047,0,1,A
333651,2,1,A
620004,3,3,B
910745,0,0,B
837662,1,1,B
943322,1,0,B
907396,2,2,B
522464,3,3,B
637028,0,0,B
929821,2,2,B
831377,0,0,B
273030,,,B
978827,1,0,B
725910,0,0,B
519596,3,3,B
484731,1,2,B
344732,3,3,B
604679,1,2,B
213215,0,0,B
595850,0,0,B
286045,3,1,B
192411,2,2,B
747516,1,0,B
729803,1,1,B
266336,1,1,B
527596,,,B
515154,0,1,B
356337,1,1,B
245176,1,1,B
599492,1,1,B
713802,1,1,B
285520,0,0,B
254784,2,1,B
396954,1,1,B
918426,1,1,B
895730,1,1,B
436572,1,1,B
106052,1,2,B
880444,1,1,B
834328,1,1,B
180569,1,1,B
383651,2,1,B
547905,1,1,B
222952,1,1,B", header = TRUE, sep = ",")

You may have noticed my example data includes some missing values, this is true of my real data so I have included it as such in case this makes a difference.

I have tested the data for normality and found that the data is not normally distributed.

>shapiro.test(exampledata1$Test.1)

Shapiro-Wilk normality test

data:  exampledata1$Test.1
W = 0.8043, p-value = 4.618e-09

As there are two groups, the data is not normally distributed and there are no repeated measures I have opted to use the Mann-Whitney-Wilcoxon Test:

wilcox.test(Test.1 ~ Group, data=exampledata1)

Wilcoxon rank sum test with continuity correction

data:  Test.1 by Group
W = 597.5, p-value = 0.01603
alternative hypothesis: true location shift is not equal to 0

Warning message:
In wilcox.test.default(x = c(3L, 0L, 0L, 1L, 0L, 1L, 2L, 3L, 1L,  :
cannot compute exact p-value with ties

The reason I am not sure if this is correct is because I thought this test used ranking, and as stated "cannot compute exact p-value with ties". As my scores only consist of 0,1,2,3 I wondered whether this would effect my result as all of the scores are repeated a number of times?

Secondly, once I have done this I want to look at the difference between scores in Test 1 and Test 2 for group A and B, to do this would I just work out the difference between the scores for each participant then perform the same test?

EDIT: Thank you both for your help, I have opted to use the categorical analysis as this is more in line with my data.

I wondered if you may be able to advise on one other thing, having used Fishers Exact Test on the 2x5 table for each test I found a pvalue of 0.03 for Test 1. However I am unsure how I determine how the groups are significantly different? Any help would be appreciated.

$\endgroup$
  • $\begingroup$ Have you considered analyzing the data like a contingency table and using a chi-square or similar test to compare groups A and B in terms of the the distributions of cases among results 0, 1, 2, and 3? $\endgroup$ – EdM Aug 25 '16 at 18:09
  • $\begingroup$ Your data do not need to be normally distributed to apply t-tests or ANOVA. Indeed, no data like these can be anywhere near normally distributed since they can attain so few distinct values. Because the variability is so limited and you appear to have a lot of data, classical (Normal-theory) methods will work well. $\endgroup$ – whuber Aug 25 '16 at 18:19
  • $\begingroup$ Thank you both for your help, could you please have a look at my edit regarding Fishers exact test? Thank you $\endgroup$ – tomclark Aug 26 '16 at 10:28
  • $\begingroup$ @tomclark Just noticed your comment and edit. Those who have commented on a question don't necessarily get informed of further comments; you can specify a single other user to be informed per comment by using the @ sign before the username. The poser of a question gets notified of all comments on the question, and the person who posted an answer is notified of all comments on that answer. $\endgroup$ – EdM Aug 28 '16 at 19:58
3
$\begingroup$

The Fisher test is one way to test whether there are differences in the distribution of cases among cells of a contingency table. If these sample data are much smaller in numbers of cases than your "real" data then a chi-square test might be better. Neither test specifies exactly how the distribution differs. Typically a display of the contingency table is fairly self explanatory, although if you are writing this up for others then you might want to add some discussion based on your knowledge of the subject matter.

It seems that you have added a category of "missing" to your 4 numeric categories. That might not be wise. Handling of missing data is tricky, particularly if the probability of a value's being missing is somehow related to its actual value. The missing-data tag on this site has many useful posts. You might also follow the multiple-imputation tag to see if that is appropriate or helpful for your application. In your case, without imputation, I might first determine whether groups A and B differed in the frequency of missing data, then proceed to analyze the non-missing values.

If you are interested in some measure of the average difference in scores between groups A and B, and particularly if you are interested in whether there are differences between Test 1 and Test 2, or interactions between the Groups and the Tests, then you should instead follow @whuber's recommendation in a comment and just treat the values as numeric in a linear model or ANOVA. As he notes, even though your data can't be normally distributed, in practice with large numbers of cases and these types of data the normal-theory methods will generally work well enough. After thinking about this, I prefer his suggestion over mine, as you have an ordered numeric score rather than simply a set of arbitrary non-ordered categories.

Note that the default for numeric tests in R (whether lm or wilcox.test) is to ignore missing data. So you should try to understand the nature of the data's missingness, whether you proceed with contingency tables or with numeric data analysis.

Finally, note that the "Warning" from wilcox.test was just that: a warning rather than an indication that your results are incorrect. The test was OK, reporting a p-value based on a normal approximation rather than an exact p-value based on the data values. If you had 50 or more cases, your call to wilcox.test would not even have tried to calculate exact p-values. The coin package in R has a wilcox_test function that can calculate exact p-values in the presence of ties, but I see no need here for an exact p-value.

$\endgroup$
  • $\begingroup$ Thankyou @EdM, you post was very informative. I will have a look at the missing-data tag. I have marked this as solved, thank you also for the tip regarding tagging. $\endgroup$ – tomclark Aug 30 '16 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.