# Survey analysis with missing data by design

I have a survey with 400 responses looking at the satisfaction of customers with a company's service overall, as well as on various specific aspects (website, account manager, invoicing, etc.). Ratings are done on a 10-point anchored scale.

The issue is that respondents only rate an aspect if they have had any interaction with it. For instance, only 14 out of the 400 responses have visited the website, so I only have 14 ratings.

The client would like to understand what drives overall satisfaction. I'd run a Shapley Values Regression using relaimpo package in R, but I can't run any type of analysis without first handling all the missing data. Any advice on doing so?

• Aug 21, 2015 at 18:01

You could start with paper by Pokropek (2011) who describes idea of data missing by design. In such case, as described in the paper, different methods for imputation of missing data are possible.

As I understand, you have survey results of survey that consisted of several parts, where each part is a group of questions dealing a specific subject and not all customers took part in all parts of the survey. As I understand, assigning certain groups of customers to different survey parts was part of your design and was not decided by study participants (if not you can have non-random missing data issues to deal with).

Let me use synthetic example. Imagine survey that was taken by individuals divided in groups (G1,...,G5), survey consisted of different parts (P1,...,P4).

The parts are dealing with comparable subject (i.e. you can assume strong correlation between the parts, they are somehow exchangable, for example, different groups of students answer different tests on mathematics). Some groups answered the same parts, e.g. G1, G2 and G5 answered P1 (see diagram below). If this is how your study looks like, than you can use methods for test equating (see Kolen and Brennan, 2004; Von Davier et al., 2004). Equating methods let you to rescale score from test $X$, to scale of test $Y$, so that you can infer what would be the score of some individual if he took part in $Y$ test instead of $X$. Simple equating methods like linear equating use basic arithmetic tricks in matching means and standard deviations of the two tests, while more advanced methods like equipercentile equating (cf. Livingston, 2004) match empirical cumulative distribution functions, Item Response Theory based methods are also used. In R you can use equate library (Albano, 2014) that implements different basic equating methods.

This could be an option for you if you can assume different parts of survey to be exchangeable and you have common parts between different groups. In your case the assumption that different parts of survey (that deal with different aspects of customer satisfaction) are exchangeable is disputable, but still you can consider using some of these methods as they were designed for similar problems. This topic is pretty wide, so I would suggest reviewing the literature before going any further.

Pokropek, A. (2011). Missing by design: Planned missing-data designs in social science. ASK. Research & Methods, 20, 81-105.

Kolen, M.J., and Brennan, R. L. (2004). Test equating, scaling, and linking. New York: Springer.

Von Davier, A.A., Holland, P.W., and Thayer, D.T. (2004). The kernel method of test equating. Springer Science & Business Media.

Livingston, S.A. (2004). Equating test scores. ETS.

Albano, A.D. (2014). equate: An R Package for Observed-Score Linking and Equating. R package version, 2.

You could impute these missing data, but then you would have to ask yourself how to interpret them, since these are cases that are missing because the question didn't apply for them. In your problem, you could interpret such imputed values as ratings for these aspects had your respondent interacted with them.

For the imputation, there are a lot of options available in R. You can see the various packages that implement imputation methods here under the Imputation section.

I think an interesting option is the Multivariate Imputation by Chained Equation method implemented on the mice package, because it quite flexible, allowing to use information from other variables (including variables with missing data) in the imputation process and it also deals with different types of variables. Also, it would be interesting to use multiple imputation, to properly propagate the variability due to the imputation (by default the function mice() on the mice package runs multiple imputation with 5 imputed datasets). I'm not familiar with the relaimpo, but taking a quick look on its documentation, it seems it can handle multiply imputed datasets using the function mianalyze.relimp()

This seems to be a difficult case for considering imputation, given that values are not missing at random (and MCAR is the critical assumption of most imputation approaches) and the sheer number of missing values, 96% in the example you describe.

More importantly though, why try to define 'overall satisfaction' by such poorly represented response categories. I'd suggest doing separate analyses on response categories, or their clusters, e.g. taking well-represented responses and analyzing them separately, then looking at the smaller population of aspect-specific responses.

You really had a bad luck. My concern is also a concern of others about how much missing data is too much? Can current algorithms deal with missing data at 50% of their values? How about 70%, 90%? In principle, MICE can deal with large amounts of missing points, but you would expect large error terms that might turn your work down. I think that you want an imputation procedure that keeps the balance of values of the units you actually have. That is, you want the mean values you currently have. You may also want to generate some scenarios with imputed values, than take the mean of them. It will be much less arbitrary than just pick one method, I guess.

Sounds like there's way too much missing data for imputation... and no theoretical justification for imputation. What would an imputed value mean? Something like, "Well, if a person had visited the website given their age and gender, then they probably would have rated it like this." I don't find that justification very compelling.

'Fancy approach': you could consider a Heckman selection model, so one model of the probability that an aspect was rated, and another model of satisfaction given that an aspect was rated, then both models are estimated together. Or, you could consider Tobit type models, which can help us account for censoring in our data.

'Easy approach': you could just squish your data. Combine all aspect ratings together, and then include indicators (a factor) for the which aspect was rated. If a respondent rates more than one aspect, this respondent gets included for each aspect that he/she rated. Then you just specify:

overall_satisfaction ~ aspect_satisfaction*aspect_factor + other_controls


Not a perfect model by any means, but it's easy, and this gets you an estimate of the relationship between satisfaction with certain aspects and overall satisfaction. Definitely would want to check for multicollinearity issues here.