1
$\begingroup$

I want to run an online survey about people's preferences for environmental policies in different economic sectors using a Likert-type scale (strongly disagree, disagree, etc.). I have 4 sectors and 4 policies. I read somewhere that "each respondent should answer one question from each of the sectors" and that "which question from each of the sectors the respondent get must be randomized". I was wondering what is the theoretical foundation for doing so?

$\endgroup$
2
  • 2
    $\begingroup$ None, AFAIK: the basis is empirical. The first time you go to the time, effort, and expense of designing and carrying out a survey and some critic (legitimately) points out that the answers might merely reflect the deterministic selection or ordering of the questions (as psychologists have discovered can actually happen), you will appreciate the defense that randomization gives you. Moreover, the randomization immediately justifies a probabilistic model that requires no assumptions at all, enabling probability-based statistical analyses. $\endgroup$
    – whuber
    Jun 13, 2023 at 21:30
  • $\begingroup$ "each respondent should answer one question from each of the sectors". Did the person who made this recommandation give any hint as to why? (I suppose they did not, but who knows.) It might be justified doing so, or it might introduce bias. For example, it may be very hard to measure a construct (here, degree of agreement with a policy) with just one question. On the other hand, for example, asking too many questions can increase the burden on the respondent -which may have several unfortunate effects like careless responding or survey fatigue. $\endgroup$
    – J-J-J
    Jun 14, 2023 at 14:20

1 Answer 1

1
$\begingroup$

I would say that there two different reasons for the two statements:

  1. Why there should one response per person per sector: We would imagine that there is some correlation between a respondents' answers. This correlation could be structured in different ways. For example a) that a person prefers the same policy in many different sectors or b) a person prefers all policies within certain sectors. I can't tell which correlation is strongest but whoever made the recommendation that there should be one response per person per sector thinks that b) is the strongest correlation. If correlation a) was stronger, one should recommend to ask each respondent one time about each policy. By asking only one time about each sector, the goal is to get less correlation between a person's answers and thereby get higher power than if one asked a person 4 questions randomly distributed between the four sectors (for example 2 in sector X, 1 in sector Y and Z and 0 in sector W). It does beg the question why one wouldn't ask each person 4 questions, one within each sector and one about each policy...
  2. Why randomizing the questions: the main reason to randomize the questions is to make sure that the order and combination of questions doesn't bias the results by acting as confounding variables. There could be something psychologically that makes people who rates policy A in sector X more likely to approve policy B in sector Y if asked right after each other. By randomizing, any such strange effects would be absorbed in the statistical model as increased variance - instead of being bias.
$\endgroup$
5
  • $\begingroup$ As for 1, we can only speculate without more information. It could just be that the person making the recommandation is making a mistake and did not even think this through. $\endgroup$
    – J-J-J
    Jun 14, 2023 at 12:45
  • $\begingroup$ Thanks for your answer. There may be some confusion regarding the answers we expect to receive: one person will be answering 4 questions in total: one for each sector. Within each sector, we randomized the policy option, so there is only one policy within each sector that the respondent has to answer (using Likert scale). $\endgroup$
    – olbap79
    Jun 14, 2023 at 13:04
  • 1
    $\begingroup$ @J-J-J, are there good arguments for recommending "one response per person per sector" other than the reducing variance? $\endgroup$
    – svendvn
    Jun 16, 2023 at 18:23
  • $\begingroup$ Off the top of my head, I can't think of one. My comment was about the fact that this is often very difficult, not to say impossible, to measure the degree of agreement with a policy with just one question. So at first glance, I'm quite skeptical of this study design, but it may be that I just don't have enough information. $\endgroup$
    – J-J-J
    Jun 16, 2023 at 19:08
  • $\begingroup$ However, one possibly misguided reason could be to reduce the burden put on respondents, by reducing the number of questions they have to answer to. Not a bad idea in general to reduce the burden, as it can impact answer quality and long term willingness to answer surveys -but this is not a good idea if you end up with an incorrect measure of what you're interested in. $\endgroup$
    – J-J-J
    Jun 16, 2023 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.