3
$\begingroup$

A test (for example, psychological) should be designed in a manner to get valid information, i.e. it must be valid instrument. Is there any relevance of reliability in constructing a test, too? What is the difference between reliability and validity? They both seem to be often based on correlations, so it is easy to mix the concepts.

$\endgroup$
  • $\begingroup$ This question was re-worded by me to some extent to make it more consistent. I believe the question deserves some upvoting to above "zero level" of votes because it happened to get some upvoted answers already. $\endgroup$ – ttnphns Sep 19 '17 at 10:14
  • $\begingroup$ @ttnphns, done. I will delete this comment later. Probably all the comments related to that discussion should be deleted. $\endgroup$ – mpiktas Sep 25 '17 at 7:48
7
$\begingroup$

Reliability has to with being able to replicate results. So if you apply the same measurment technique multiple times in similar situations you should get similar results. An unreliable measurement adds a lot of random noise to your measurement.

Validity has to do with measuring what you want to measure. This is a much more theoretical concept: if this is a survey you just need to think about what the questions are, how a respondent might interpret that question (and the possible answer categories), the theoretical concept you want to measure, and whether these all match up.

Also see here and here

$\endgroup$
  • $\begingroup$ validity is not a theoretical concept simply. It can be assessed and checked ie cross-validated $\endgroup$ – Subhash C. Davar Jan 7 '14 at 11:16
  • $\begingroup$ If I wanted to know the anual cheese consumption in the Netherlands and I measured the number of ants per square meter in Berlin, then that would not be a very valid measure of annual cheese consumption. How would cross-validation help us to detect that? $\endgroup$ – Maarten Buis Jan 7 '14 at 13:11
  • $\begingroup$ You can always deceive a naïve person with statistics. A statistician hopefully understands the meaning of "target population" and thus takes a sample from the similar or equivalent population. $\endgroup$ – Subhash C. Davar Jan 10 '14 at 17:44
  • 3
    $\begingroup$ This is not about the difference between statisticians and the rest nor is this just about sampling from the right population. Validity is about measuring what you want to measure, and getting the sample wrong is just one way in which you can fail to achieve that goal. More common is to just ask the wrong question or a question that is misunderstood by the respondents. Looking at your data (with more or less advanced statistical techniques) is unlikely to definitively show whether or not you have a problem. Most often you will just need to sit down and think about it to determine it. $\endgroup$ – Maarten Buis Jan 11 '14 at 13:54
  • $\begingroup$ I believe one should examine the same or equivalent population even if we have different settings. Cross validation does not imply that we should not account for given structure. $\endgroup$ – Subhash C. Davar Jan 11 '14 at 16:55
7
$\begingroup$

Validity of a psychometric scale is its correlation with the trait supposed being measured (called criterion), and uncorrelation with extraneous systematic factors.

Reliability of a psychometric scale is its correlation with its own "true" score, and uncorrelation with extraneous random factors.

This is what @Maarten said, put more formally.

Often seen as two different facets of an instrument functioning, reliability and validity sometimes are difficult to separate conceptually from each other. As seen for example with the term "internal consistency"$^1$. They seem to be two different (and perhaps competitive) paradigms about correlations observed among the items constituting the scale. Reliability stance believes that the measured trait is simple, unifactor structure; each item measures it and just it + is liable to some random error. The lesser is the error the more item-item homogeneous is the scale. Validity stance regards the overall score by the scale as if it were the external criterion; validity is interested in that item-to-total correlation, rather than in how well items duplicate each other. Thus, what the reliability approach will tend to pre-include into the error term, the validity approach will ascribe to the complexity, the multi-aspect, multi-symptom nature of the trait (and decide whether it is still the same construct or is already a battery of different ones). If "systematic error" were easy to distinguish from "random error" without assumptions, there would be no frictions between reliability and validity.


$^1$ Cronbach's alpha is one of measures of internal consistency from reliability standpoint.

$\endgroup$
  • 2
    $\begingroup$ Thanks Sir!,I am agree with your answer.Basically reliability is authenticity of a construct /Test/Questionnaire while validity is the degree of trust worthiness ,soundness and accuracy of data that gives information out of it( test/questionnaire). $\endgroup$ – Subhash Chander Paul Jan 4 '14 at 8:07
  • $\begingroup$ I guess that this is a problem with English, but I would say it the other way around. $\endgroup$ – Maarten Buis Jan 4 '14 at 9:44
  • $\begingroup$ @ttnphns Though your answer is clear to a person who has an idea of psychometry and statistics. It is comprehensive. can CFA (confirmatory factor analysis) help us compute validity coefficient. $\endgroup$ – Subhash C. Davar Jan 7 '14 at 10:55
  • $\begingroup$ Generally, yes. It depends on how you use CFA and what is a the criterion for you. $\endgroup$ – ttnphns Jan 7 '14 at 11:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.