I'm having an argument with a colleague (neither of us have a strong statistical / research methods background) on whether we should randomize questionnaire layout.
The survey in question is a feedback forms on teacher performance which are distributed for high school pupils. So one pupil gets up to 20 surveys (on each teacher he has contact with) which have identical, multiple-choice questions. Most of the questions also have identical scale. The pupil fills the surveys in online and does not necessarily need to do everything at once (but he definitely can). We do not have control how / when they choose to do it.
My colleague argues that by randomizing the order of questions we reduce the possibility that pupils will fill in the questionnaires without thinking too much - e.g. if one fills 5 forms at once, the evaluations of teachers will be very similar.
In my opinion, though, we would actually gain from making sure that the survey questions are grouped together based on the area of question (e.g. teacher communication skills, usage of IT, etc) - this would ensure that pupils provide consistent answers.
I guess there's a point in both approaches - but maybe there are some best practices how to deal with that?
Oh, and if somebody could dig up some references, that would be ultra-cool. My colleague likes facts :)