# How to assess whether experimental measurements obtained from different technicians are biased?

Suppose I have a list of measurements from an experiment; for example,

34 31 55 18 19 22 44 48 23 . . .

But I then learn that these experiments were conducted by two different technicians, so I embolden those conducted by tech #1, and I leave untouched the measurements of tech #2:

34 31 55 18 19 22 44 48 23 . . .

I want to know if the bold numbers are somehow biased relative to the unbolded numbers. How should I measure this?

My intent is to compare the mean and variance of the overall sequence to the means and variances of each of the subsequences. But I'm a novice at statistics, so perhaps this naive approach is too shallow?

• Actually, this is related to one of the deepest (and, in some sense, most troubling) problems in classical statistics. See the Behrens-Fisher problem. – cardinal Oct 1 '11 at 21:37
• @Cardinal, I read the wiki page (and I am by no means a statistician). Actually I am even more confused. Under the assumption of normal distribution, wouldn't the problem boil down to a t-test? And if they don't come from a normal distribution, still they come from the same distribution. In that case, isn't there an equivalent to z-score possible? – Arun Oct 2 '11 at 2:15
• @Arun, Sorry if my comments confused matters. It was meant more as a side note. If the two samples are independent, and each comes from a normal distribution with the same variance, but possibly different means, then, yes, a two-sample $t$-test is the natural choice. There are strong theoretical reasons for its use. The interesting thing from a statistical perspective is that these theoretical properties break down in a big way as soon as we let the variances be unequal. In applications, this is often seen. :) – cardinal Oct 2 '11 at 3:22
• @Cardinal, thank you, that clears things a bit. Can we mathematically be certain of the population variances being equal? I mean, how does one go about validating the assumptions of a t-test before using one. Or this is normally a determinant of experiment? – Arun Oct 2 '11 at 10:10

This answer is probably a bit late for you, but I think this falls under the category of repeatability. If anyone disagrees, feel free to comment!

Repeatability is the extent to which the identity of the observer can be used to predict the result, or in simple terms, the consistency with which one observer differs from another.

Repeatability can also be used to quantify how closely repeated measurements of an individual resemble each other, relative to measurements from another individual. (e.g. when measuring the weight of a bird at several occasions over a year, high repeatability means that the individuals consistently differ across the course of a year).

Repeatability is quantified by the intra-class correlation coefficient (ICC). In your case, the class is the observer. R can be calculated by ANOVA, or the best-practice method, linear mixed effects modelling (LMM).

If you have a low repeatability within observers (small R, nonsignificant p-value), you can say that the observers are not biasing the result.

If you have high repeatability, you have a problem, although it is not so bad if the observer error is balanced across treatment groups (but still should be avoided if possible). This would perhaps be a good time to use a model in which the observer is specified as a random factor, so that adjustments can be made for the inter-observer differences.

Here is a very good review on the topic by Nakagawa and Schielzeth (2010): http://onlinelibrary.wiley.com/doi/10.1111/j.1469-185X.2010.00141.x/abstract

And the related R package, with support for general and generalized linear mixed modelling: http://rptr.r-forge.r-project.org/

The first thing I would do would be to plot the data.

But then, as you say, compare the means and SDs for each technician, to each other. A t-test would be the first thing to try, for comparing the means.

• Could the two technicians have the same mean and variance yet be measuring differently? – curious_cat Apr 2 '13 at 7:50