3
$\begingroup$

Okay, I think this might be a basic question but I just can't quite seem to get there. I understand what reliability and validity is, but I don't understand just what I should do with it when looking at the results.

Imagine a super-simple test examining how much people spend on groceries every week.

For example, say that that test gives a significant result that is reliable but with low validity. So I get that that means that the test wasn't good for looking at the thing we wanted to look at, but that we would get roughly the same results if we repeated it. What can I do with that result then? Does it make the result completely useless? Anything else than draw the conclusion that we can't say anything about the subject that we wanted to look at?

The opposite as well: a valid test with low reliability. Anything else than drawing the conclusion that we can't say anything?

I'd be very grateful for any help. Thank you.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

A reliable result that has low validity can't really be redeemed. In your example, it would be like recording something silly like people's height to determine how much money they spent instead of just recording how much money they spent.

An unreliable result that has high validity can be redeemed because it's a problem of measurement error. In your example, suppose you made a numerical mistake in converting currency. In less silly examples, you can integrate measurement error in your model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you. So a follow up question: how does validity and reliability affect whether we get a significant result or not? Say that we're doing a study and looking for a correlation between two things, and we get a significant result: if we find out that our method had high reliability but low validity, how did that affect the result? Or the other way around: same test, significant result but with high validity but low reliability. What effect did that have on the test? I'm suspecting that it should decrease the chance that we get a significant result? Thank you for your help. $\endgroup$
    – user36688
    Dec 30, 2013 at 12:13
  • $\begingroup$ Sorry for the late response. I am going to edit my answer above within 24 hours to include an example that will illustrate your concerns. For now, I don't think there is a universal answer yes/no that applies. As far as I know, in real life problems, models have measurement errors integrated into the before doing statistical tests. This is because you know you're making a measurement error since you're not able to measure the thing you want to measure. E.g. in animal studies, you may not be able to measure because it would either kill the animal or require you to remove it from its habitat $\endgroup$
    – rocinante
    Dec 30, 2013 at 18:52
1
$\begingroup$

Here are some basic facts might help you think about this topic.

  1. Utility is the impact of the use of a test, usually computed in financial terms in comparison to not using the test. The formula for utility depends on validity, not reliability. The formula also depends on other factors, not all easy to estimate. In short, utility is proportional to validity (not the square of validity).

  2. Low validity tests can have practical utility. For example, improving the productivity of the workforce by a small amount (say 1% or 3%) might mean many millions of dollars in savings to a large employer.

  3. The validity of a test can be no greater than the square root of the reliability.

To answer your question, a test with low validity might be useful in some circumstances.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.