Establishing that the population sampled of two separate surveys is the same

Question

I have two surveys of two separate populations (I don't know that they are necessarily distinct, but they are from two different databases) that ask a similar set of questions. Some questions are basic demographics (e.g. age, income), while other questions are a bit more detailed or about their opinions (e.g. brand preferences, spending habits).

How do I prove statistically that the two populations are "the same," or at least comparable? I know that I can do a t-test for individual questions, but is there a way to establish similarity on more than one dimension?

The goal is to combine the surveys from these two populations into one series of survey data. For example, we may run survey A every six months, but we run survey B every month, except when we run survey A. I would then like to combine the results from survey A and survey B to have a monthly series of survey data.

Answer 1

I think that for subject-specific characteristics, like demographic data, you can proceed the usual way (t-test, etc.). This will help showing that your samples don't differ according to these variables. About self-reported attitude data, if you have very few items, skip to step 2, otherwise step 1 might be appropriate.

1. Assessing measurement equivalence

Rather than saying that the two populations (or actually, samples) are "the same", I would say you have to show that your two questionnaires are assessing the same construct(s). This is what is done in cross-cultural surveys or international clinical trials where health-related quality of life is used as a secondary endpoint, for example. In each case, we have a set of items that purports to assess different dimensions, and we want to demonstrate whether we are measuring individuals in the same way irrespective of their country. When dealing with uni- or multidimensional scales, it is know as measurement invariance in psychometrics, that is you want to show that the factorial structure is comparable between the two groups. But, the same remark would apply as well if we were considering longitudinal data (I interpret your question as involving different samples at each time point). A multi-group confirmatory factor analysis is appropriate in this case. Standard references include:

• Meredith, W (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525-543.
• Vandenberg, RJ and Lance, CE (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-70.

In R, the lavaan package provides facilities for that kind of analysis, but see the documentation: lavaan: an R package for structural equation modeling and more (§6.2). Otherwise, you have to resort on Mplus or a good software for SEMs. Studying Measurement Invariance Using Confirmatory Factor Analysis provides illustration with LISREL syntax.

You may want to consider data from 6 to 12 months (to collect 1 or 2 waves for survey A). After that, I think you can just pool your data.

2. Assessing group comparability

Now, if you cannot define a clear construct common to those two questionnaires, or if you have so few items that it would make no sense to consider a scale, then you can rely on basic group statistics for each item (using e.g., t-test, trend test for ordinal data, tests for nominal data, etc.). In this case, you are essentially studying between-group differences. This basically tells you whether (aggregated) scores differ, but not whether items are perceived as having the same meaning (or underlying the same construct) across the two groups.

Answer 2

To add to chl's answer, another step you can take to ensure your data is representative of the population as a whole is to compare both samples to a third party data set. In the United States, there is the American Community Survey which I often use to compare the data I work with to the population for a given region. The other thing to consider is whether or not your data should match up with ACS data (or whatever you are comparing to). For example, the data I work with often relates to estimating travel demand and so I am interested only in the "traveling" population for a region. The ACS data samples from a broader range than the "traveling" population, so some differences may be expected.