What is the difference between reliability and validity? A test (for example, psychological) should  be designed in a manner to get valid information, i.e. it must be valid instrument. Is there any relevance of reliability in constructing a test, too? What is the difference between reliability and validity? They both seem to be often based on correlations, so it is easy to mix the concepts.
 A: Reliability has to with being able to replicate results. So if you apply the same measurment technique multiple times in similar situations you should get similar results. An unreliable measurement adds a lot of random noise to your measurement.
Validity has to do with measuring what you want to measure. This is a much more theoretical concept: if this is a survey you just need to think about what the questions are, how a respondent might interpret that question (and the possible answer categories), the theoretical concept you want to measure, and whether these all match up.
Also see here and here
A: Validity of a psychometric scale is its correlation with the trait supposed being measured (called criterion), and uncorrelation with extraneous systematic factors.
Reliability of a psychometric scale is its correlation with its own "true" score, and uncorrelation with extraneous random factors.
This is what @Maarten said, put more formally.
Often seen as two different facets of an instrument functioning, reliability and validity sometimes are difficult to separate conceptually from each other. As seen for example with the term "internal consistency"$^1$. They seem to be two different (and perhaps competitive) paradigms about correlations observed among the items constituting the scale. Reliability stance believes that the measured trait is simple, unifactor structure; each item measures it and just it + is liable to some random error. The lesser is the error the more item-item homogeneous is the scale. Validity stance regards the overall score by the scale as if it were the external criterion; validity is interested in that item-to-total correlation, rather than in how well items duplicate each other. Thus, what the reliability approach will tend to pre-include into the error term, the validity approach will ascribe to the complexity, the multi-aspect, multi-symptom nature of the trait (and decide whether it is still the same construct or is already a battery of different ones). If "systematic error" were easy to distinguish from "random error" without assumptions, there would be no frictions between reliability and validity.

$^1$ Cronbach's alpha is one of measures of internal consistency from reliability standpoint.
