I have a large sample of 800 participants who completed a measure of relationship at 2 time points. The result of a paired-samples t-test indicates that there was no statistically significant difference between both time points. I then split the group into high risk and low risk subsets of participants likely to develop relationship problems over time. the analysis then indicated a statistically significant result for both subset! I do not think that this is correct as I feel that one subset should be significant relative to the other? am I right in thinking this or is there a better way of splitting the group into subsets rather than my approach of creating a dummy variable for each subset and selecting each subset and carrying out the paired samples t-test twice?
1 Answer
I feel that one subset should be significant relative to the other?
From what you posted it doesn't sound like that's the comparison you were making but rather that you were comparing time t1 with time t2 in both subgroups. If that is the case I don't see any reason why the subgroups can't show a significant difference even if the overall population did not.
Assuming that your null hypothesis is that the mean difference between t1 and t2 is 0 it's not hard to imagine situations that might produce such a result: if the high-risk group has greater risk at t2 relative to t1 and the low-risk group has lower risk at t2 relative to t1, the overall group difference between measures at t1 and t2 could be close enough to 0 that you would not reject the null. If you are unsure about this you can always run some simulations with random data to see if this situation arises-- I threw one together in R using rnorm and found 10 such cases out of 10,000 trials.
The method for grouping your subsets seems fine to me, though it's possible that your criteria for subsetting the data could cause an issue (depending on how you categorize people as high- or low-risk).
-
$\begingroup$ Thank you very much for your insight. It just didn't make sense to me. the overall group performed worse at time 2, albeit non-statistically significant reduction in scores p = .88. The subset considered to be at high risk of developing relationship problems at time '1' (n = 31) reported an improvement in scores at time '2' p = <.001, r = 0.8, while the subset that were considered to be low risk at time '1' (n=245) reported a reduction in scores at time '2' p = .01, r = 0.2. From reading your post, i get it makes sense as one subset improved and the other did not. $\endgroup$ Commented May 20, 2016 at 19:48