2
$\begingroup$

Assume a simple example: A survey asks people how much they spent in the last week in total amount and in individual days. As a hypothetical response, one may answered that the Total amount = 10 and every single day (Sun to Sat) = 1. So they are not the same.

What should we do if we find out this difference? E.g., in the data cleaning process in which we sum individual measures to see if it matches withe the total amount. Let's also say that these numbers quite make sense, i.e., they fall within the expected/acceptable range.

$\endgroup$
2
$\begingroup$

To answer this question, you should think about the possible reasons of why the two ways of reporting give different results. Ideally, you should ask each individual respondent what they meant when they gave two different answers. This is rarely possible.

Some ideas. Perhaps your definition of a "week" was different to the one of the participants (e.g. Sun to Sat vs Mon to Sun). It's also possible that they thought about specific days, but a typical (as opposed to the last) week, since this is a more abstract category than individual days. A rounding error might have occurred: only rough estimates were given for daily spending and the uncertainties accumulated. They might not have remembered their spending for a specific day, but paying closer attention for weekly allowance, memorized their spending better.

Finally, you should keep in mind that the participants might not have cared about such discrepancies; as a participant, I have answered with non-compatible answers, myself. Some information might be hidden in the divergency, perhaps giving you feedback on the quality of your questions. A valid question (thanks to @Tim for hist comment) to ask yourself might also be whether the discrepancy occurs for all participants or just some individuals. If the latter is the case, can that tell you anything about them?

Another important aspect to the question of what to do with it is what you intended to measure. Is it average daily spending for the week? Then it's probably reasonable to average the individual day-answers, rather than dividing weekly spending by 7. Is it average weekly spending? Then this is an estimate in any case, so an option might be to average the sum and the weekly-response.

Thinking about your specific wording of the questions should help you decide what reason is the most plausible for different results and guide you in resolving the disparity.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I would add: does the particular discrepancy tell something about individual participants? $\endgroup$ – Tim Apr 16 '15 at 8:12
  • $\begingroup$ That's a fair point, @Tim. Added to the answer. $\endgroup$ – Fato39 Apr 16 '15 at 8:17
  • $\begingroup$ @Fato39 Thanks for your thoughtful input. As the data cleaning is often times post-hoc, there are few chances to follow with participants for correction (my particular issue is not this one, just use it as example). When discrepancies occur, we are left to decide which to use, for which I think your proposed strategy/reasoning for selection between the two (intention of measurement and use) sounds interesting. I'd be glad if you have further, formal references not necessarily to support your specific point but as well to provide guidance on how to address problems of this kind and others. $\endgroup$ – NonSleeper Apr 17 '15 at 7:10
  • $\begingroup$ @NonSleeper I think this is fundamentally a psychological (specifically, psychometric), rather than purely statistical question. If you ask people the same question on multiple occasions, you are bound to get different answers - this concept is called reliability. Asking about facts does not change much, since you are still indirectly measuring recollection of those facts. I cannot think of any references that would deal with your specific problem, unfortunately. If you find one, please, let me know. $\endgroup$ – Fato39 Apr 17 '15 at 9:55
  • $\begingroup$ @NonSleeper Elaborating on reliability, this is a somewhat related problem. As an example, consider this happiness scale. It asks about general happiness and then more specific aspects of it. Which answer (they are different in general) merits a larger statistical weight? Most often, in the field of psychometric this question is resolved with simple averaging of the answers (individual "items" of a scale). Discrepancy in the answers in considered a kind of a reporting noise. $\endgroup$ – Fato39 Apr 17 '15 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.