If a measure is valid (but not necesarily reliable), can it be consistently replicated?

Question

I had this question on an exam, and I was positive that the answer was A.

The Beck Depression Inventory is a scale intended to measure depression levels, with higher scores indicative of higher levels of depression. If this were a valid measure of depression, we would expect that:

A) the results of the inventory cannot be consistently replicated.

B) a person's score on the inventory is not related to his or her level of depression.

C) people who get higher scores on the Beck Depression Inventory are more depressed than people who get low scores.

D) people who get lower scores on the Beck Depression Inventory are more depressed than people who get high scores.

I don't think my answer was wrong. If the test was valid, it is not necessarily reliable. Just because you are taking the same test, you are not going to get the same score every time. You are going to get a score that reflects your depression level at the time of taking the test. Which means that the results cannot be consistently replicated.

Answer 1

On multiple choice exams you're supposed to pick The Right Answer. (C) is definitely correct as all it says is that the test is valid - using other words. So if you know what validity is, you should pick (C). Anything you might say about (A) depends on a number of interpretations and assumptions -- it is not the most unambiguous option I've seen but it's not too bad either provided that one uses the minimum amount of common sense.

But your reasoning about (A) is not based on common sense. Although one may interpret the words "consistently replicated" as a requirement that the measurement results should be exactly numerically precisely the same every time, from now until the end of the world as we know it, this is almost certainly not what is meant when anyone uses these words. In other word, stating that the results can be "consistently replicated" does not mean that the results are "perfectly reliable". This may be a question of nuance, if you're picky, but that's how these words are used.

Another problem with your interpretation of (A) is that it does not use the information given in the question text. The text says that the inventory is valid. On the other hand, for saying that an inventory is not "perfectly reliable" (i.e., your interpretation of "consistently replicable") one needs no information about the inventory whatsoever. There is no inventory that is "perfectly reliable" -- except the one that spits out the same score every time.

Thirdly, you say, "Just because you are taking the same test, you are not going to get the same score every time. You are going to get a score that reflects your depression level at the time of taking the test." But here you're mixing the stability of a trait (depression) with the reliability of a measure (BDI). Perfect reliability implies that a person should get the same score provided that his or her depression level has not changed. This is why test-retest (or any other method we can use with inventories) is not a pure measure of reliability -- in real world, there is always a fluctuation of trait levels.

Answer 2

The answer is (C), because that is exactly what the question stated: Higher scores indicate higher levels of depression. If this is a valid measure, then those who score higher are in fact more depressed than those who score low.

Your reasoning for (A) is not correct. A valid measure is not necessarily reliable, but more importantly, a valid measure does not imply it must be unreliable, which is what (A) states. In essence, you are misinterpreting the word "cannot" to mean "might not."

Answer 3

Validity and reliability are different, but not orthogonal conceptions. Validity is not equal to "unbiasedness", albeit some people actually state that it as unbiasedness. (To me, validity is sensitivity or precision in measuring a right thing.)

But no, everything that rises must converge, and completely valid instrument is then automatically completely reliable too. Really, if it correlates perfectly with what it is deemed to measure (the criterion) then there is no room left to correlate with disturbing factors. With moderate correlation with the criterion, there is such a room. And then the question arises: to what extent these disturbing factors are "random" and to what they are "systematic". Correlation with random ones are the unreliability side and correlation with systematic ones are the invalidity side of a test.

Time (test-retest) - we usually consider it as random factor ("random factor" in not statistical, but in psychometric sense) and thence test-retest is reliability dimension. Sometimes, though, we might choose to consider time as systematic factor (i.e. a "misleading" alternative criterion which we want our test not to measure); then test-retest will be validity dimension.

So, validity and reliability are - to my mind - sooner two distinct paradigms or approaches to deal with extraneous correlatedness, than two real, ever separate qualities of a test.

Answer 4

A reliable measure is one that consistently produces the same result when measuring the same thing. Therefore reliability also limits validity: a non-reliable test measures not that which it is designed to measure, but also random noise.

So to answer the question: C. If the test (claims to give a higher score when someone is more depressed) and (the test is valid) then a higher score on the test means that someone is more depressed.