8
$\begingroup$

I have 2560 paired observations from an experiment in which participants provided two ratings for a set of objects, at two different points in time. Half of the objects in the set had the value of an attribute A changed between the two time points, half did not. Of the objects that were changed in each participant's set, half went from A' to A'' and half from A'' to A'. (i.e. all participants experienced both orders). My main hypothesis is that changing this attribute from A' to A'' leads on average to a higher rating, and this is indeed supported by the data. I am also interested in determining whether the magnitude and perhaps direction of this effect depends on the A' rating.

For the purposes of this question, I am considering only those instances where A was changed (1280 pairs of obs). The following GLMM

(A'' rating - A' rating) = participant + order + A' rating

where A' rating is a covariate and participant and order are categorical variables, leads to the conclusion that there is a significant positive correlation between A' rating and effect of changing to A'' and that this correlation is <1, such that objects with a low A' rating have their rating increased by changing to A'' but that objects with a high A' rating actually get rated lower when changed to A''.

I want to test whether this is simply due to regression to the mean. To this end, I have followed Kelly and Price in using Pitman's test of equality of variances for paired samples and would appreciate some feedback on whether I've done the right thing.

This is what I did, following the suggestion of a colleague:

1) calculated the SD of A'' ratings $(SD_1)$ and the SD of A' ratings $(SD_2)$
2) regressed A'' rating on A' rating and recorded the correlation $r$.
3) calculated T as $T=\frac{\sqrt{(n-2)} [(SD_1/SD_2)-(SD_2/SD_1)]}{2 \sqrt{(1-r^2)}}$

The 2-tailed p value of T (Student's t dist with 1280-2 DF) is 0.07, i.e. at alpha=0.05 there is no significant difference between the variances for the two sets of ratings and thus no effect of A' rating on rating difference beyond regression to the mean. (We can argue about 2-t vs. 1-t p values later).

I now plan to adjust my difference scores to account for this and re-do the GLMM outlined above, as outlined by Kelly & Price.

If you've got this far through the detail, then firstly well done, and secondly, can you tell me if 1) my colleague's suggestion was sensible and 2) if it was, have I implemented it correctly? I have a couple of concerns/apparent grey areas but I'd be interested to hear what others have to say first.

Thank you.

$\endgroup$
1
  • 1
    $\begingroup$ Please may I suggest a new tag along the lines of "artefacts"? (Or "artifacts" if you happen to be of the American persuasion). $\endgroup$ Sep 9, 2010 at 15:06

2 Answers 2

3
$\begingroup$

As far as I know, the Pitman test is formulated as :

$$F=\frac{SD_2}{SD_1} ~with~ SD_2 > SD_1$$

$$T=\frac{(F-1)\sqrt{n-2}}{2\sqrt{F(1-r^2)}} $$ with $r$ the correlation between the scores in sample 1 and sample 2. This is not equivalent to the formula you use and mentioned in the paper. I'm not positive about my formula either, I got it from a course somewhere (alas no reference...)

Apart from that, it might be interesting to take a look at an alternative approach to dealing with regression to the mean. I found the tutorial paper of Barnett et al on regression to the mean very enlightening.

Now let's get back a moment to the 2-sided versus 1-sided p-values. Regardless of the formula you use, the sign of T is only dependent on the order of the SD's. (In fact, how I know the pitman test, T is always positive.) Hence, the underlying distribution is -as far as I'm concerned- not the T distribution but half the T distribution, meaning you have to put the cutoff at $T_{0.975, df}$, but the related p-value is originating from one tail only. This is equivalent the standard F test for comparing variances.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks. I've done some more searching and your formula appears to be more commonly used (and using a ratio of variances seems simpler and more intuitive to me). I tried both formulas on two data sets; the results were qualitatively similar in one case and quantitatively identical in the other, so I guess the results of the two versions of the test converge at some point. I'm going to proceed using your formula. If anyone else has any insight into the two formulae, it would be great to hear it. $\endgroup$ Sep 13, 2010 at 9:01
2
$\begingroup$

the short answer to your question is that with a sample size of n = 1280, a formal statistical test for equal variances is not really necessary. with that sample size, the variance 'estimates' you get are certainty-equivalents [almost certainly very close to the true population variances]. so you can just look at the two values and see if they look reasonably close or not.

a comment about your linear model: i assume there is a random error term in it as well. i think there should also be an intercept term, unless you are sure the regression line goes thru the origin.

also, do you treat the participant effect as a random effect? it seems you should, if each contributes more than one pair of ratings to the data. but that might cause problems with the usual methods for testing for effects other than a regression effect [for which the usual models do not have such terms]. this is because the usual models assume the different before-after [ratings] pairs are independent. but the random effect creates correlations between the ratings pairs for the same individual. such a model would have to be studied further to see if the hypothesis of no differential A'-rating effect still reduces to a test of equal variances.

here is a bit more about the pitman test [altho what follows is probably unnecessary for you, in view of the 'short answer' above]:

like tests for equality of variances for two [or more] independent samples, the pitman test is apparently sensitive to the non-normality of the responses. you don't mention the scale on which the variable rating is measured, but especially if it is a likert-type scale, the assumption of normality for the responses may be questionable.

wilcox did a study of the pitman and other tests of variance equality for paired data and wrote:

But, it is concluded that, in terms of controlling both Type I and Type II errors, a satisfactory solution does not yet exist. [this was in 1990.]

this doesn't necessarily mean that the pitman test would be invalid for your data, but it does suggest that some caution in using it might be appropriate. [if i remember correctly, the difficulty with homogeneity-of-variance tests arises when the data distribution [for rating in your case] has a heavy tail [or tails]. then the actual level of the test can exceed the nominal level [of .05, say], resulting in too often falsely rejecting the hypothesis of equal variances when, in fact, they are equal. [better power, but also worse false rejection rate.]

it seems unlikely that responses on a likert scale would have heavy tails [or too big a 4$^{th}$ moment - as compared with its variance], but it might not hurt to see if the usual kertosis measure [4$^{th}$ central moment divided by variance$^2$ exceeds 3 [which is then an indication of heavy tails].

grambsch in 1994 compared the pitman test to alternative procedures - some of which are more complicated to carry out.

some numerical results in her paper suggest that if the distributions of rating are uniform or have the shape of a half-parabola over the range of responses [i.e. have light tails], the pitman test is conservative: the actual type 1 error for a nominal value of $\alpha = .05$ can be anywhere from .005 to about .05 - depending on the actual shape of the distributions and the sample size. [her results deal with samples sizes of 50 or less. i suppose with your sample size of 1280, her conclusions would have been somewhat different.]

if you are interested, she presents a modification of the pitman test that keeps the actual significance level close to the nominal value of .05, for both light- and heavy-tailed distributions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.