Where is the boundary between repeated measures and non-repeated?

Question

We're currently planning an experiment and wondering whether this should be regarded as repeated measures. (And more generally, where the boundary between repeated measures and non-repeated measures is.)

In the experiment, subjects are asked to find defects in two work-products (in counter-balanced order). One of four treatments has been applied to these work-products, and we are interested in differences between the treatments. The number of defects is the dependent variable.

We assume that the number of found defects depends on the subject and therefore that each subject's result for the first and second work-product are correlated, which would indicate repeated measures. But the number of found defects probably also depends on the work-product, so in a way we have different measurement instruments and therefore no repeated measures. So should we use a test for repeated measures?

Taking this a bit further, we consider measuring some covariates per subject, build a regression model and analyze the residuals (ANCOVA?). This will decrease the correlation between subject and defect residuals. So if it was repeated measures in the first case, will it still be repeated measures here, too?

Answer 1

Yes, these are repeated measures of each subject. This causes the dependency you have to consider.

Now you say that the number of found defects depends on the work-product. I think this dependency concernes only the expectation on each work product, not the stochastic dependency between two measurements as each work-product receives only one treatment, right?

If a work-product receives all treatments successively, you will get a more complicated dependency structure, so it is still repeated measures.

Now concerning regression elements. Ideally, the right regression model will in fact remove the dependency of the residuals in normally distributed data. But typically, you don't know the right regression model and your data may only be approximately normal, so some dependency remains.

Answer 2

There is no question that these are repeated measures, since each subject evaluates two items.

If the evaluation is highly objective then then the correlation between the two measurements might be zero or near zero.

Think of asking raters to classify objects as blue or red. If you present objects that are clearly bright red and bright blue, then all the raters make the same judgements and there is no variation among raters. If you present rates with a wide variety of colors, puce, lavender, bright blue, bright red, some shades of orangey-brown and some of purplish-brown, then the background experience of the raters might have a strong effect on their decisions and you will find that raters vary greatly. This gives a strong effect of rater and a larger component of the variance.

I myself have no fashion sense and have relatively few color words that I can use reliably. My daughter, on the other hand, uses a wide variety of color words reliably. I would expect her to make different red-blue distinctions than I do.

When you get to further analysis you might need to use ANOVA for incomplete blocks since each rater is evaluating items with two of four possible treatments.