4
$\begingroup$

Suppose I have a list of measurements from an experiment; for example,

34 31 55 18 19 22 44 48 23 . . .

But I then learn that these experiments were conducted by two different technicians, so I embolden those conducted by tech #1, and I leave untouched the measurements of tech #2:

34 31 55 18 19 22 44 48 23 . . .

I want to know if the bold numbers are somehow biased relative to the unbolded numbers. How should I measure this?

My intent is to compare the mean and variance of the overall sequence to the means and variances of each of the subsequences. But I'm a novice at statistics, so perhaps this naive approach is too shallow?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Actually, this is related to one of the deepest (and, in some sense, most troubling) problems in classical statistics. See the Behrens-Fisher problem. $\endgroup$
    – cardinal
    Commented Oct 1, 2011 at 21:37
  • $\begingroup$ @Cardinal, I read the wiki page (and I am by no means a statistician). Actually I am even more confused. Under the assumption of normal distribution, wouldn't the problem boil down to a t-test? And if they don't come from a normal distribution, still they come from the same distribution. In that case, isn't there an equivalent to z-score possible? $\endgroup$
    – Arun
    Commented Oct 2, 2011 at 2:15
  • 1
    $\begingroup$ @Arun, Sorry if my comments confused matters. It was meant more as a side note. If the two samples are independent, and each comes from a normal distribution with the same variance, but possibly different means, then, yes, a two-sample $t$-test is the natural choice. There are strong theoretical reasons for its use. The interesting thing from a statistical perspective is that these theoretical properties break down in a big way as soon as we let the variances be unequal. In applications, this is often seen. :) $\endgroup$
    – cardinal
    Commented Oct 2, 2011 at 3:22
  • $\begingroup$ @Cardinal, thank you, that clears things a bit. Can we mathematically be certain of the population variances being equal? I mean, how does one go about validating the assumptions of a t-test before using one. Or this is normally a determinant of experiment? $\endgroup$
    – Arun
    Commented Oct 2, 2011 at 10:10

2 Answers 2

Reset to default
3
$\begingroup$

This answer is probably a bit late for you, but I think this falls under the category of repeatability. If anyone disagrees, feel free to comment!

Repeatability is the extent to which the identity of the observer can be used to predict the result, or in simple terms, the consistency with which one observer differs from another.

Repeatability can also be used to quantify how closely repeated measurements of an individual resemble each other, relative to measurements from another individual. (e.g. when measuring the weight of a bird at several occasions over a year, high repeatability means that the individuals consistently differ across the course of a year).

Repeatability is quantified by the intra-class correlation coefficient (ICC). In your case, the class is the observer. R can be calculated by ANOVA, or the best-practice method, linear mixed effects modelling (LMM).

If you have a low repeatability within observers (small R, nonsignificant p-value), you can say that the observers are not biasing the result.

If you have high repeatability, you have a problem, although it is not so bad if the observer error is balanced across treatment groups (but still should be avoided if possible). This would perhaps be a good time to use a model in which the observer is specified as a random factor, so that adjustments can be made for the inter-observer differences.

Here is a very good review on the topic by Nakagawa and Schielzeth (2010): http://onlinelibrary.wiley.com/doi/10.1111/j.1469-185X.2010.00141.x/abstract

And the related R package, with support for general and generalized linear mixed modelling: http://rptr.r-forge.r-project.org/

Improve this answer
$\endgroup$
1
$\begingroup$

The first thing I would do would be to plot the data.

But then, as you say, compare the means and SDs for each technician, to each other. A t-test would be the first thing to try, for comparing the means.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Could the two technicians have the same mean and variance yet be measuring differently? $\endgroup$
    – curious_cat
    Commented Apr 2, 2013 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.