Regression or regression alternatives for data missing not at random

Question

I am analyzing the results of a satisfaction survey.

The research question or goal is to determine whether or not (dis-)satisfaction with performing certain activities (as measured by a 7-point satisfaction scale) on a system better predicts or explains overall satisfaction with the system (also measured on a 7-point satisfaction scale) than other activities.

The tricky part about the analysis is that not every participant provided a satisfaction rating for each activity (predictor). They were only asked to rate their satisfaction with an activity if they also self-reported performing using the system to perform that activity. This is leading to a lot of missing data.

In other words, imagine the three predictor variables are X Y and Z. The survey structure was essentially:

Q1. Have you used system ABC to do X? [Yes / No]

Q2. Have you used system ABC to do Y? [Yes / No]

Q3. Have you used system ABC to do Z? [Yes / No]

The participant would only receive the satisfaction (7-point scale) question for activity "X", "Y" or "Z" if they selected "Yes" to the corresponding questions above.

There were about 6 activities. Of the approximately 1900 participants, only around 150 provided a satisfaction rating for every single activity - so, with listwise deletion, only about 7% of the sample remains. This, to my knowledge, is such a significant loss of data that techniques like multiple imputation are just not feasible -- coupled with the fact that I expect my missing data would be classified as "missing not at random".

Having said that, there are still a significant number of data points for each predictor variable and the outcome variable - no less than 500 for each - it's only that it's extremely rare for any 1 participant to provide ratings for all of the predictors.

I feel that ordinal regression may simply be inappropriate and there may be no way to really resolve this problem if I want to include all or most predictors. However, if I'm mistaken I welcome any feedback.

What methods might be best suited for exploring how well satisfaction ratings with these activities best predicts overall satisfaction with the system given that each record may not have data for several predictors? I've read loglinear analysis may be a possible approach -- but I'm not familiar with that analysis. Alternatively, I've considered just doing basic correlations -- but this doesn't really compare the activities against one another in a model, of course.

Thanks in advance.

Answer 1

I am currently working on a very similar problem.

I have worked out two solutions as most viable and I will use both to compare results and choose the best.

1) Use missing values as information

Not doing a certain activity is information in itself, so instead of coding it as missing it could be coded as another value (e.g. 0 = Not participated; 1 = Participated and not satisfied [e.g. 1,2,3 on a likert scale]; 2 = Participated and satisfied [e.g. 4,5 on a likert scale]) and consequently all variables can be used in a regression.

2) Using methods more robust to missing data

Classification And Regression Trees (CART) cope generally very well with missing data and can be used for any problem of classification/prediction.

What doesn't work very well:

Imputation methods might work but in my experience missing values because of skipped questions by design are very difficult to impute because they are logically not MAR (data isn't missing at random, it's because respondents choose not to participate in activity X). This means that by design respondents not participating in activity A are all very similar (e.g. older people generally abstain from mobile banking) and therefore no values in the sample could be used as a valid basis for imputing the missing values.

Answer 2

This problem might be well-suited to hierarchical linear modelling, such that satisfaction is nested in activities. The way you describe your data, it might be organized in a "wide format:"

Participant  Satisfaction on Activity X  Satisfaction on Activity Y  ...  Satisfaction on Activity j
1            1x                          1y                          ...  1j
2            2x                          2y                          ...  2j
...          ...                         ...                         ...  ...
i            ix                          iy                          ...  ij

The first step is to reformat your data into a "long format:"

Participant  Activity  Satisfaction
1            x         1x
1            y         1y
.            .         .
.            .         .
.            .         .
1            j         1j
2            x         2x
2            y         2y
.            .         .
.            .         .
.            .         .
2            j         2j
.            .         .
.            .         .
.            .         .
i            j         ij

In this format, missing data is less of a problem. With $overall$ as your outcome variable, $participant$ indexed as $i$, and $activity$ indexed as $j$, the hierarchical linear model would be:

$Level 1:$

$overall=\beta_0+\beta_1(satisfaction_{ij})+e_{ij}$

$Level 2:$

$\beta_0=\gamma_{00}+\gamma_{01}(activity_j)+U_0$

$\beta_1=\gamma_{10}+\gamma_{11}(activity_j)+U_1$

And the mixed model would be:

$overall=\gamma_{00}+\gamma_{01}(activity_j)+\gamma_{10}+\gamma_{11}(activity_j)(satisfaction_{ij})+e_{ij}+U_0+U_1$.

For more information on this type of model, see Raudenbush & Bryk (2002).