Unpaired repeated measures data

Question

I realise this is one of those "Don't start from here" type questions, and I did indeed tell them not to "start from here" but it would be a shame to junk the data.

Basically I have some interval normal data, Likert scale type stuff, from two groups, at time one and time two, and I know that some subjects in time two are the same ones as from time one, so really the data should be paired. However, the data was collected anonymously so we just have two groups to compare.

My best thought would be that I could report a very conservative estimate of difference and then say "for all the problems with data collection, this is the smallest the effect is likely to be".

Edit: added-

If significance is possible too, i.e. "and I am reasonably confident this effect is nonzero" then, great.

I don't even know where to begin looking up such a stupid way of analysing data. Would bootstrap be useful at all?

Any suggestions gratefully received, as I say highly conservative ones I think would be fine.

Answer 1

A simple two-sample t-test would actually be a conservative way of testing for time differences here.

Let's say that $t_{paired}$ is the t-statistic that you would get from a paired t-test on your data, if you knew the pairings, whereas $t_{unpaired}$ is the t-statistic you get from a simple two-group t-test on the data. These two t-statistics have the following relationship: $$ t_{paired} = \frac{t_{unpaired}}{\sqrt{1-r}} $$ where $r$ is the correlation between the time 1 scores and the time 2 scores, which you can't estimate because you don't know the pairings in the data.

$t_{paired} = t_{unpaired}$ when and only when $r = 0$. But notice that this is also exactly the condition where $t_{paired}$ takes its minimum value$^1$. As $r$ increases, $t_{paired}$ becomes larger and larger compared to $t_{unpaired}$.

So if you just report the two-sample t-test, essentially assuming that $r = 0$, then what you are really doing is reporting a lower bound on the correct $t_{paired}$. If you have a significant difference according to the two-sample t-test, then the paired t-test would definitely also show a significant difference if you could compute it. But if there is not a significant difference according to the two-sample t-test, it is still possible that the difference would be significant if you could compute the paired t-test.

---

$^1$ Technically, theoretically, $r$ could go down to as low as -1 and this would actually be the minimum value of the function across the range of $r$. And when $r<0$, the paired $t$-test is actually less powerful than the unpaired $t$-test. However, in real life it would be unlikely (without further info about the specific context) for the time1 and time2 scores to have a negative correlation. It is even unlikely for the correlation to be 0. There is most likely a modest positive correlation.