7
$\begingroup$

I realise this is one of those "Don't start from here" type questions, and I did indeed tell them not to "start from here" but it would be a shame to junk the data.

Basically I have some interval normal data, Likert scale type stuff, from two groups, at time one and time two, and I know that some subjects in time two are the same ones as from time one, so really the data should be paired. However, the data was collected anonymously so we just have two groups to compare.

My best thought would be that I could report a very conservative estimate of difference and then say "for all the problems with data collection, this is the smallest the effect is likely to be".

Edit: added-

If significance is possible too, i.e. "and I am reasonably confident this effect is nonzero" then, great.

I don't even know where to begin looking up such a stupid way of analysing data. Would bootstrap be useful at all?

Any suggestions gratefully received, as I say highly conservative ones I think would be fine.

Improve this question
$\endgroup$
  • $\begingroup$ If you are just interested in the effect size (i.e., the standardized mean difference) the question of paired versus unpaired data does hardly influence your results. Pairing gives you only an advantage in terms of significance testing. So what do you really want to know? $\endgroup$ – Henrik Aug 19 '13 at 13:19
  • $\begingroup$ If you can't be sure of the membership of the groups, you can't dismiss the possibility of biases due to uneven sampling, e.g. some people got bored, or died, or whatever and didn't return. On the other hand, there are lots of cases in which that's true (most opinion polls, for example?) so methodology for this case would be of wide interest. $\endgroup$ – Nick Cox Aug 19 '13 at 18:52
11
+50
$\begingroup$

A simple two-sample t-test would actually be a conservative way of testing for time differences here.

Let's say that $t_{paired}$ is the t-statistic that you would get from a paired t-test on your data, if you knew the pairings, whereas $t_{unpaired}$ is the t-statistic you get from a simple two-group t-test on the data. These two t-statistics have the following relationship: $$ t_{paired} = \frac{t_{unpaired}}{\sqrt{1-r}} $$ where $r$ is the correlation between the time 1 scores and the time 2 scores, which you can't estimate because you don't know the pairings in the data.

$t_{paired} = t_{unpaired}$ when and only when $r = 0$. But notice that this is also exactly the condition where $t_{paired}$ takes its minimum value$^1$. As $r$ increases, $t_{paired}$ becomes larger and larger compared to $t_{unpaired}$.

So if you just report the two-sample t-test, essentially assuming that $r = 0$, then what you are really doing is reporting a lower bound on the correct $t_{paired}$. If you have a significant difference according to the two-sample t-test, then the paired t-test would definitely also show a significant difference if you could compute it. But if there is not a significant difference according to the two-sample t-test, it is still possible that the difference would be significant if you could compute the paired t-test.

$^1$ Technically, theoretically, $r$ could go down to as low as -1 and this would actually be the minimum value of the function across the range of $r$. And when $r<0$, the paired $t$-test is actually less powerful than the unpaired $t$-test. However, in real life it would be unlikely (without further info about the specific context) for the time1 and time2 scores to have a negative correlation. It is even unlikely for the correlation to be 0. There is most likely a modest positive correlation.

Improve this answer
$\endgroup$
  • $\begingroup$ Sorry it took so long to award the bounty, I've been caught up with work. Thanks very much for this helpful answer, well worth 50 points. In fact it's inspired another question which I will post as soon as I get chance. $\endgroup$ – Chris Beeley Aug 28 '13 at 19:01
  • 1
    $\begingroup$ No problem @ChrisBeeley. By the way, I wanted to mention a small correction to something I wrote in my post. I said that the minimum value of $t_{paired}$ occurred when $r=0$. But technically, theoretically, $r$ could go down to as low as -1 and this would actually be the minimum value of the function across the range of $r$. However, in real life it would be exceedingly rare and strange for the time1 and time2 scores to have a negative correlation. It is even unlikely for the correlation to be 0. There is most likely a modest positive correlation. So I think my reasoning above is still sound. $\endgroup$ – Jake Westfall Aug 28 '13 at 19:28
  • $\begingroup$ Oh yes, good point, I hadn't thought of that. But yes I quite agree that is a theoretical consideration and extremely unlikely in real life. I might go and have some fun now trying to think of things that negatively correlated at Time 1 and Time 2. $\endgroup$ – Chris Beeley Aug 29 '13 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.