-2
$\begingroup$

I'll try to explain my question with an easy example.

Very often you can read in the papers that researchers have compared the IQs of different countries. But, it's a known fact that the intelligence tests are standardized in each country so that the average is 100. How is it possible, then, to compare the IQs of different countries, if each country will have an average of 100?

The intelligence is just used here as a catchy example, my question is more broad and concerns comparing different groups in variables that have been standardized.

E.g. let's say that you have data from previous independent researchers who have measured a personality trait in two countries. They used the same questionnaire, however, the researchers from the first country used a 4-point Likert scale, while the researchers from the second country have used a 5-point Likert scale. Obviously, the "raw" scores cannot be compared to infer if the countries differ in the measured trait. However, if we standardize the scores, we will also be unable to compare the countries because their average would be exactly the same.

So, how would we go on about this issue?

$\endgroup$
1
$\begingroup$

If the tests are truly equivalent, then you should be able to compare standardized scores in one country with those in another. (I would want to make sure that the options leading to various Likert values are equivalent also. And that standardization was done equivalently in both countries.)

Equivalent tests, overlapping populations: Suppose student A takes a college admissions test with scores distributed $\mathsf{Norm}(100, 20),$ scoring 130, and that student B takes a an equivalent college admissions test with scores distributed $\mathsf{Norm}(150, 30),$ scoring 195.

To be sure that the two tests are equivalent, we would probably need data from many students who took both tests. (In the US, this might be feasible to do for two types of admissions tests, both of which have been taken by many of the same students. There is a lot of data about the equivalence of these tests. My own university accepts scores from either test, and has tables for fairly converting one kind of score to a satisfactory approximation of the other. For an example, see these equivalence tables.)

The respective z-scores of students A and B are $Z_A = \frac{130 - 100}{20} = 1.50$ and $Z_B = \frac{195-150}{30} = 1.5.$ Each student scored 1.5 standard deviations above the mean on whichever test they took. Then it seems fair to conclude that A and B are equally adept at taking such tests.

Same test, different populations: By contrast (against all real-world evidence), let's imagine people in country A have very different intelligence levels than people in country B. And that they take exactly the same intelligence test. Then if the data on means and standard deviations used to standardize the scores in each country came only from that country, it would be difficult to compare.

If the z-scores of a person from A and a person from B are both 0, then we can only conclude that each person is of average intelligence for his/her own country, but it would not be possible to say they are are of equal intelligence to each other. Thus, if z-scores are to be compared across countries, it is important that universal data be used to standardize all scores.

$\endgroup$
  • $\begingroup$ Completely understandable, but what if I don't want to compare individuals, but countries and the countries used a same questionnaire with the different length of Likert scale? $\endgroup$ – J. Doe Aug 29 '18 at 22:20
  • $\begingroup$ I don't think it will work. I don't see that you have a conversion link from one Likert scale to the other. To make matters worse, a 5-point scale probably has a 'neutral' option, whereas a 4-point scale doesn't. My limited experience is that the availability of a 'neutral' response fundamentally changes the answering process. Are the languages the same in the 2 countries? Any chance of getting a fairly large diverse group of individuals to take both tests (half with one first, half with the other first) to try to determine a rough equivalence? $\endgroup$ – BruceET Aug 29 '18 at 22:26
  • $\begingroup$ Unfortunately the languages are different and there is no possibility for me to get samples of participants to take those tests because I am not the one conducting those studies - instead, I gained access to online databases with studies that were already conducted and I wanted to compare different countries on some variables. $\endgroup$ – J. Doe Aug 30 '18 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.