0
$\begingroup$

I'm currently working on cleaning/preprocessing a bunch of survey data from a collection of similar but distinct surveys. In order to combine the survey results this involves, among other things, identifying common (as in shared) questions (which might, obviously, not be identically phrased).

While the data set is small enough[1] to do this manually, it got me curious about what models/methods might be appropriate for automating this. Some quick searches revealed nothing, though this might just be do to not knowing the appropriate terminology.

I was thinking about applying a clustering method on the collection of all questions, using some similarity measure taking into account both the survey answers (which can be of a number of different types, though in this case primarily open ended text, multiple choice, and 1 to 5 scale) and the question text. The obvious problem with this is that each survey contains a large number of similar questions (e.g. a slew of "How much do you like _ on a scale 1 to 5").

My question is therefore whether this approach can be salvaged (for example by first identifying question types and then matching them between surveys) and/or whether there are other, more appropriate, solutions.

EDIT/CLARIFICATION: My problem is one of finding similar/same questions among the surveys. For example, while multiple surveys contain a question asking for year of birth, these questions might be worded differently (e.g. "Born:" vs. "Year of birth:" vs. "When were you born?").

EDIT/ADDITIONAL BACKGROUND: The surveys are simple questionnaires, all sent to the same set of schools. The exact wording of the questions have changed over time (the survey spanning multiple years) and might differ depending on the exact audience (students, teachers, management e.t.c.).

All this being said, I will most probably do this manually, and my reason for asking is mostly curiosity, so feel free to relax/add assumptions.

[1] About 1400 "questions" in 16 surveys, though most of these are "artifacts" of the way the survey software exports multiple choice questions, the real value being something like 10 to 20 questions per survey.

$\endgroup$

2 Answers 2

1
$\begingroup$

This is really going to depend on what format the questionnaires and/or codebooks are in. If they're in a standardized, ideally text, format, you could consider parsing the codebooks (e.g., in R with something like what is described here), which might allow you to easily identify and match common questions.

In practice, however, I find this is almost never easy. When question wordings and codings change, so do variable names and the formats of codebooks. Take a look, for example, at the inconsistency over time of the American National Election Studies data, which use generally standard questions but alternate between text and PDF codebooks, all arranged in different ways.

When I've had to combine multiple questions from different surveys, I generally work in the following workflow:

  1. Establish the set of constructs/variables you want to have in the final data set and put all of these into a spreadsheet (i.e., as separate rows).
  2. Go through all of the surveys and line-up each survey's question identifier with the relevant construct.
  3. Decide how you want the data in your final dataset coded and establish the set of recoding procedures to convert all of the original surveys into the desired format.
  4. Merge all of the identified questions from each survey into a single dataset.

In short, parts of this could be automated, but it all depends on what format the original questionnaires/codebooks are in.

$\endgroup$
3
  • $\begingroup$ I'll accept this answer, though I was curious about any possible (semi-)automated methods (I've already finished this manually). A problem with your suggestion in this particular case would have been that I'm not the one to decide what construct/variables are of interest (I was simply to process the data "losslessly"). What I ended up doing wasn't all that different though. I continually partitioned the set of answers into classes using hand crafted regular expression until each partition contained only "one question". $\endgroup$ Jul 21, 2013 at 22:29
  • $\begingroup$ Don't feel obligated to accept. The regular expression approach is probably your best bet unless you want to get into a more sophisticated natural language processing algorithm...but I suspect at the scale that you're working, developing that will be more time consuming than doing it manually. It's a great problem/question, but I suspect the best answers relate to data producers releasing better metadata rather than end-user solutions for dealing with poorly aligned surveys. $\endgroup$
    – Thomas
    Jul 22, 2013 at 6:53
  • $\begingroup$ While not exactly what I was after it's still sound advice, and a reasonable answer to the question as it was posed. $\endgroup$ Jul 23, 2013 at 11:26
0
$\begingroup$

The usual way to match two surveys is to match across the common variables for example, demographic variables like race, gender. To put in simple words, there are two different surveys, one measures only the income (plus the demographic variables) and another only the expenditurse (plus the demographic variables). Now, you can combine these two surveys on the basis of common variables so that you can understand the relationship between income and expenditure. See, Statistical Matching: Theory and Practice for details.

Note: The matching is achieved through harmonization of definitions of common variables.

$\endgroup$
5
  • $\begingroup$ It looks like I didn't formulate my question clearly :), sorry about that. My problem is about actually identifying which questions correspond to each other between the different surveys, e.g. one survey might formulate the question simply as "Gender:", while another might say "What gender do you identify yourself as:". That is to say, it's more of a text/string-matching problem. $\endgroup$ Jun 21, 2013 at 14:20
  • $\begingroup$ What do you want to achieve from that?I am not aware of that type of study in my field. $\endgroup$
    – Metrics
    Jun 21, 2013 at 14:22
  • $\begingroup$ It is a question of combining survey answers from multiple surveys written at different times, where the creators have changed their wording from year to year (though they all target the same population). $\endgroup$ Jun 21, 2013 at 14:25
  • $\begingroup$ you mean an effect of time series variation in the survey questions on the response; should be an interesting topic. $\endgroup$
    – Metrics
    Jun 21, 2013 at 14:33
  • $\begingroup$ Well, while it does sound interesting what I'm after is much simpler :). I'm interested, primarily, in the change in response over time, not so much as an effect of the working. Also change over time is only one aspect, the surveys may also differently formulated for different audiences (specifically students, teachers, and management), and we wish to generate statistics for responses on the population as a whole (i.e. students, teachers, and management together). $\endgroup$ Jun 21, 2013 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.