Should item nonresponses be removed before calculating Welch's t-test?

Question

I performed a survey in which one of the questions used a grid to let people express preferences for different features of something, using a Likert-like scale for each feature. I coded the values as 1-5. The people surveyed fall into two disjoint subgroups; one subgroup had more individuals in it. Here's an illustration of the results (the real survey had more people and more questions). The columns are individual people's responses:

group1 = {
 "feature1" : [5, 1, 4, 5, 2, 5, 5, 4, 4, 2],
 "feature3" : [3, 5, 2, 3, 4, 2, 0, 2, 1, 2],
 "feature2" : [3, 3, 2, 1, 2, 3, 3, 1, 2, 4],
}

group2 = {
 "feature1" : [5, 4, 5, 0, 4, 3],
 "feature2" : [1, 4, 1, 0, 4, 4],
 "feature3" : [2, 5, 3, 4, 3, 2],
}

In a few cases, sometimes a person didn't give a rating for a given feature. This is represented as 0's in the lists above. For example, the 4th person in group 2 didn't give responses for features 1 and 2 and only responded to feature 3 (here, with a rating of "4"). I have no way to know why people occasionally did that. (Not always the same person, not always the same question.)

Question: If I want to do Welch's unequal variances t-test for the responses of the two groups to each feature, how should I treat the item nonresponses? In concrete terms, if I use Python SciPy's stats.ttest_ind and pass it two lists of values (one for each group's responses to a given feature question, or in other words, for a row in the two dictionaries of responses shown above), then which of the following is the correction approach?

1. leave the 0's in the lists of values passed to stats.ttest_ind
2. remove the 0's from the lists of values passed to stats.ttest_ind

On the one hand, I can argue that removing the 0's is correct because they are not actual responses—they are just place holders we used to code the results. On the other hand, if the 0's are removed, then it seems like it would change the sample mean for one or the other group in the t-test calculation, because there would be fewer responses than there are people in the subgroup.

I don't know enough statistics to know whether what's right. Can someone please explain what the right thing to do is in this situation?

Answer 1

'0' is an arbitrary label you've used for non-response. Treating it as a number would imply that non-response expresses less satisfaction than any of the other values on the scales—as you've said you don't have any idea why people didn't respond, that hardly seems sensible. What if you'd coded non-response as '99'?

Mapping '0' to any of other the values on the scale is similarly problematic. @Carl suggests mapping to '3', citing a study that claims it makes little difference whether the 'midpoint' on a Likert scale is labelled 'undecided' or 'neutral': I can't see how that bears on the matter of how to treat non-responses (moreover—is '3' in any sense a midpoint in your Likert-like scale?). Potential reasons for someone's not responding might include their never having used that feature, not understanding the question, skipping the item by mistake; so it's a courageous decision to equate it with selecting a response that, after all, they could have selected—indeed were prompted to select—had they wanted to. You'd have to be pretty sure you could tell why people didn't respond, & that it made no odds to lump them in with people who did pick some particular answer, at least for the purposes of your study. Better in general to concede simply that they didn't tell what they think & see where you can go from there.

Excluding non-responses raises two questions. First, what's the population you're interested in making inferences about? Unfortunately it's unlikely to be just people who respond to survey questions about a feature. But it may well be just people who are familiar with a feature, so no harm in excluding people who aren't, if that's who the non-responders are. Second, are non-responders typically going to have different opinions to the responders (within each group)? Probably not when they skip an item owing to a momentary distraction; possibly when they can't follow the question; probably when they think they'll get into trouble if they say they're not keen on a feature.^†

I'll note in passing that it's good practice to use feedback from a pilot study to tweak the design to minimize rates of non-response, & to follow up some non-responders to investigate further.

Of course, you know your survey better than me, so you'll have to decide what assumptions you can sensibly make to justify excluding non-responses, if you can at all. That you say there are "a few" cases is grounds for optimism. A sensitivity check of some kind is advisable: a simple, though stringent, one would be to set all the non-responses to 1 for Group 1 & 5 for Group 5, then vice versa; & see just how much difference it makes to your conclusions. It might be a good idea to compare non-response rates between the two groups, if that can help test your assumptions.

A more complex approach is to impute the missing values. If you can predict someone's response to the item on Feature 2, say, fairly well from their response to the items on Features 1 & 3, then you've a way to fill in the blanks for non-responders. The assumption here is that the non-responders on a particular item are typically not going to have different opinions to the responder given their responses to other items. You'd probably need to do some studying or get some help if you wanted to go down this route.

† To be a little more technical, casewise deletion is appropriate when responses are Missing Completely At Random or Missing At Random with respect to a model's predictors (here just 'Group'): no biases are introduced into estimates, & imputation (based on those same predictors) would add no extra information. The early chapters of Little & Rubin (2014), Statistical Analysis with Missing Data introduce these ideas.