How to treat illogical survey responses

Question

I have submitted a survey to a sample of artists. One of the question was to indicate the percentage of income derived by: artistic activity, government support, private pension, activities not related with arts. About 65% of the individuals have replied such that the sum of the percentage is 100. The others don’t: for example, there are who answers that 70% of their income derived by his/her artistic activities and 60% by income government, and so on. My question is: how should I treat these observations? Should I delete, modify or keep them? Thank you!

Answer 1

This is a good situation for a sensitivity analysis. Analyze your data in each of three ways --

1. As they are
2. After excluding "the illogicals", i.e., people whose percentages don't add up to 100 (or 100 +/- 10)
3. After adjusting where necessary so that each person's percentages add up to 100

Then compare results, sharing any rationale you can develop as to which results might be more accurate, or more accurate in certain respects.

You can also investigate the range of ways in which the logicals and illogicals differ, if any. Do the illogicals tend to report higher incomes? To show greater support for certain ideas or programs? To skip more questions? To evince more bias in the sense of straightlining or disproportionately choosing middle responses or extreme responses?

With about 400 or these illogicals, you have enough data even to assess the relationship between degree of illogicality and degree of a given type of bias. Something like a dose-response relationship.

What you learn from these investigations might be fed back into your plan for dealing with the illogicals when it comes to the main analyses of interest.

Answer 2

As already alluded to here, those answers aren't necessarily illogical. For example, you say

The others don’t: for example, there are who answers that 70% of their income derived by his/her artistic activities and 60% by income government, and so on.

That makes perfect sense if 30% of income is derived from artistic activities done for the government. Then we really have three groups:

• Artistic activity, unrelated to government: 40%.
• Government, unrelated to artistic activity: 30%.
• Artistic activity subsidized by government: 30%.

Those numbers add up to 100%.

Consider the following questions:

• Are food stamps government or activities not related to the arts?
• Is Social Security government or a pension?
• Is a government pension (former government employee) a private pension or government?
• Is a government grant to paint a painting artistic activity or government?
• Is a job teaching arts at a local school government or artistic activity or activities not related to the arts?
• If retired from a job in the arts (teaching or commercial, e.g. drawing gift cards), is the pension private or artistic activity? Or in the case of teaching, possibly government?

You seem to want these categories to be mutually exclusive. However, I don't think everyone would interpret them that way. You may have clear thoughts on how those activities should be categorized, but it's not clear that your respondents had the same divisions in mind when they answered. At minimum, if you want the numbers to add to 100%, you should tell people that.

Personally, I think that the best approach to this kind of a problem is to do a kind of focus group. In a traditional survey, you may not be able to validate answers. So call or visit people who would be targeted by the survey and start a conversation. Then when they give answers that you don't understand, ask them why. And beyond that, ask them how you should have asked the question so that you'd get the kind of results that you want. This works more like a focus group in that it is interactive.

Once you've done that, then you can get a better idea of how to handle the responses that don't fit into your format. For example, you might take the 30% extra and subtract half from each. Then you'd have 55% artistic activity and 45% government. Or you might recategorize as 40% private artistic activity, 30% government-sponsored artistic activity, and 30% other (in this case, government support unrelated to artistic activity, e.g. food stamps or rent support). Or throw out the survey and redo it, because people are not understanding your categories properly. Part of this depends on what you understood the categories to mean as well as how they interpreted it.

Too late now, but for future surveys, consider doing a regular focus group before the survey. Then you can test your questions in the group environment and improve them. You may even find that you get additional questions from the group. If it's too difficult to do this in person, consider doing it online. Or do a test survey (of a smaller number of people, validating responses with follow-up questions) by phone personally before doing the real survey. Any of these can help you make your questions clearer.

For example, perhaps your real categories should have been private income from artistic activity; government-sponsored artistic activity; private pension from a former job; other income unrelated to artistic activity. Or something different. Part of the problem is that I can't tell what you want, which makes me think your respondents couldn't either. If there are three different interpretations, it's almost like you are mixing responses from three different surveys.

Answer 3

I cannot give you an answer for the general case of illogical responses. But for this specific type of question - been there, done that. Not only in a survey, but also in semistructured interviews, where I had a chance to observe how people come up with this kind of answer. Based on this, as well as some general experience in observing and analyzing cognitive processess, I would suggest: normalize your data back to a sum of 100%. The reason is that people seem to first go to the most salient category - in your case, that would be the largest income - give a gut-feeling estimate for it in percent, then start thinking of the next smaller categories and base their estimate relative to the anchor of the first category, plus that of further already mentioned categories.

For an example, a train of thought will go like: "My first source of income is certainly more than half. It makes what, 60%? No, that's too low, let's say 65%. The second is about a third of that, so that would be a bit more than 20%, uh, difficult to calculate it in my head, let's round up to 25%. The third also feels like a third of the first, but it is actually always a bit more than the second, so it should be 30%. Or even 35? No, let's go with 30. Oh, and I forgot that I have a fourth source, that only happens once a year, that should be really small compared to the others, so 5 or 10%? Probabaly 5 is closer, it isn't really that much". And so you end up with an answer of 65 + 25 + 30 + 5 = 125%.

Because people tend to be more aware of the relative size of the income parts to each other than of each part to the total, I would say that normalizing them is in order here, if you want to run some kind of numeric analysis on the income. I would only work with the actual reported numbers if the difference between people's beliefs and statements about their income and objective reality is an important topic for your work, for example if you are a psychologist studying cognitive biases, or if you are more interested in the self-perception of artists than in their economic circumstances.

Sadly, I don't have a good literature source to prove that it really works as I described it, it is just my personal empirical observation. But I don't think that reviewers will get caught up on this kind of decision, since, as the other answers said, there is no single "right" way to treat it. If anything, they will dismiss your whole data from this question as invalid due to a flawed querying technique. The best you can do is to preemptively acknowledge it and come up with arguments why your work is nevertheless useful and why the conclusions you are drawing are still good despite this specific source of inaccuracy in the data.

Answer 4

If social science has taught me anything, it's that if you give people a chance to give logically inconsistent responses, they will. So rest assured that there's nothing unusual about your subjects. This is something to keep in mind for designing future surveys. For the time being, it may be best to leave the responses as-is and just keep in mind in your analyses that the responses won't actually add up to 100%, as one would think. Rather than true proportions, you have noisy signals of how much income each subject gets from each category, so analyze them that way.

Answer 5

artistic activity, government support, private pension, activities not related with arts

Just at a glance, it seems that "artistic activity" and "activity not related with arts" should add up to 100%.

Of course "activity not related with the arts" is not the same as "not activity related with the arts," since there can be income associated with no activity at all. But that's hair splitting that most artists won't notice.

If you assume that categories 1 and 4 should add up to 100%, and reinterpret those respondents accordingly, you may find that most of them have included categories 2 and 3 in with category 4.

However, all of this is data manipulation that is not ideal. If you want accurate statistical answers, you must collect data that is accurate. People might lie in response to your survey, and that's hard to guard against, but if people honestly trying to answer your questions can be in confusion over what is meant, your survey needs to be rewritten.

Next time proofread the survey for understandability, as well as ambiguity, before you send it out.

Answer 6

You have given four categories for income - what about income that is in none of them? For example, dividend income from holding stocks. It's not income from any form of activity, yet it's not government support or a pension either. I would suggest that in the absence of other information you should regard the responses as correct and attribute the missing money to sources that the respondents did not consider to be covered by the categories.

Answer 7

Actually that is quite simple (and not even as illogical as you might think)! I assume that it is really the proportions of the different categories of income you are after by asking for percentages. So you can simply renormalize to 100%. In your example: if somebody says: 70% of my income is from artistic activities and 60% is from government support, this person (who has propably never had any training in working with percentages) is actually saying: the relative sizes or proportions of my income from artistic activities and government are about 70 to 60 or 7 to 6 (propably not realizing that percentages are supposed to add up to 100..). You can convert these statements about proportions to statements about percentages by simply renormalizing them, as follows: 70 / 130 * 100 = 53% artistic income, and 60 / 130 * 100 = 47% government support income..

(what I do here is actually take 130% as a "new" 100% and calculate the proportions..)

PS. this works for all cases where the sum of the stated percentages is unequal to 100

Hope this helps!

Answer 8

The majority of answers already given have already provided some insight into the obvious survey methodological flaws, so I will not dwell on that here. Instead, I'll provide a few practical options on how to treat this data given that it's already collected and despite the flawed question. There are a few ways to handle this. You could consider marking responses that did not meet your definition of a "valid responses" by treating the entire question as missing and then following any number of practices for handling item-nonresponse such as those discussed here.

You might also consider scaling each response so that the percentages add to 100. Assuming each response is recorded as a percentage, this can be done by re-coding each original response $y_{{old}_{ij}}$ $(j=1,2,3,4)$ of the 4 sub-components to your question (i.e. artistic activity, government support, private pension, activities not related with arts) into a new response $y_{{new}_{ij}}$ as follows:

\begin{eqnarray*} y_{new_{ij}} & = & \begin{cases} 0 & ,\,\text{for}\,\sum_{j=1}^{4}y_{old_{ij}}=0\\ \frac{y_{old_{ij}}}{\sum_{j=1}^{4}y_{old_{ij}}} & ,\,\text{for}\,\sum_{j=1}^{4}y_{old_{ij}}>0\\ Missing & ,\,Otherwise \end{cases} \end{eqnarray*} So for example, say you had a respondent $i$ who answered as follows:

 A. (i=1) Artistic activity:  10% 
 B. (i=2) Government support: 0% 
 C. (i=3) Private pension: 30% 
 D. (i=1) Activities not related with arts:  40%

Then you'd recode as follows:

\begin{eqnarray*} y_{new_{i1}} & = & \frac{10}{10+0+30+40}=\frac{10}{80}=12.5\%\\ y_{new_{i2}} & = & \frac{0}{10+0+30+40}=\frac{0}{80}=00.0\%\\ y_{new_{i3}} & = & \frac{30}{10+0+30+40}=\frac{30}{80}=37.5\%\\ y_{new_{i4}} & = & \frac{40}{10+0+30+40}=\frac{40}{80}=50.0\% \end{eqnarray*}

Note that all the new percentages now add to 100%. Whatever you do, please be sure you make any transformations very clear when reporting your results and I think @rolando2 provided some excellent advice on how to perform some sensitivity analyses to see how transformations like these might affect your conclusions.