Techniques for MNAR without Imputation

Question

Oftentimes I see that people address missingness from the perspective of missing at random (MAR). However, there are cases where missing not at random (MNAR) in which one can sometimes be handled with stuff like imputation. However, I'm thinking of another case which is more complex but likely not uncommon.

Let's say participants are recruited from a study which had two years of recruiting participants. In the first year, a survey was administered with 20 questions. However, the second year required 30 questions because ceiling effects became very evident. Now if one was to run an analysis using these two years, there would be obvious issues because one year's participants did not get the full range of questions.

What would be the appropriate way of handling this data? Should one only use the 20 questions used for both years? Should the person running the analysis impute the data? What are ethical ways of handling this data? Given it would be substantial (at least 50% of the population wouldn't have answered 10 questions), I would think this would be important to address.

Answer 1

I'll make an answer out of my comments.

(1) Do not impute the 10 missing questions for all participants of the first year. Note in particular that if the reason why the 10 additional questions were asked are ceiling effects, it means that it is clear that the original 20 questions are not enough to differentiate between the high scoring participants. But this means that there is no reliable way to predict how those participants in the first year would have responded to the additional 10 questions from the data. The 20 original questions do not give enough information, and if your aim is to run a regression with speaking ability ("MNAR variable") as response and some $x$-variable as predictor, you shouldn't use that $x$-variable for imputation either, as this would mean that you assume a specific regression connection that you want to find out about in the first place. In other words, you'd make up data in such a way that it would wrongly suggest that your overall regression is more precise than it actually is, as imputation would take for granted (without having data to check) that what holds in the second year for all questions extends to the first year (I believe that in this case even multiple imputation techniques would not give you a reliable indication of uncertainty, because they still need such assumptions for the actual imputation).

Instead, assuming that your aim is the regression mentioned in the comments, I'd try out a few things with particular focus on visualisation and understanding what goes on:

(2) Looking at the original 20 questions, can any difference between the two years be detected?

(3) Regression using only the 20 original questions, with thorough visualisation to see what the ceiling effects do (potentially more sophisticated techniques than standard regression can help, here's where the answer of @Björn comes in; that could probably even address analysing all data together, but this may rely on problematic assumptions. Another option to handle the ceiling problem would be ordinal regression, which however here may require lots of data due to the large number of parameters with 20 or 30 categories.

(4) Regression on all 30 questions using the second year only.

You can then see whether all results allow more or less the same interpretation.

PS: Note that "missing at random" (somewhat confusingly) does not mean that missingness occurs randomly, but rather that the randomness of the missing values themselves can be fully explained by observed information. Whether this is the case is generally unobservable (one would need to observe the missing values to check this), and may in principle hold for your data (involving the observation year as observable information). However I agree that it is safer to treat these data as at least potentially MNAR.

Answer 2

The particular situation you describe could potentially be handled using methods for censored data. I.e. data where you know don't know the exact value, but it's greater than some value ("right censoring"), smaller than some value ("left censoring") or in some interval ("interval censoring").

However, that only makes sense, if the same question gets asked with a truncated response scale (let's say 0 to 50) and without a truncated scale (let's say 0 to 100), and if (that's a big IF) the participants interpret the question the same way each time (i.e. in the same circumstances they would answer X on the truncated scale and X on the untruncated, just when X is >50, then they respond 50 on the truncated scale). In that circumstance, an answer of 50 on the truncated scale simply means $\geq to 50$. That's exactly the situation, for which methods for censored data (here right censored) were developed.

I would be a bit nervous about doing that, because I'd worry that people would treat/interpret the scale differently, in which case it becomes very difficult to say something across the two different questionnaires.

Generic methods for censored data of course only make sense when censoring occurs completely at random or in a way that can be explained by the observed data ("at random"; of course the things influencing censoring should be included in the analysis model to deal with it, otherwise you end up in the next scenario below). If you are in a situation where it occurs dependent on the value. What do I mean by that? Let's imagine you ask people their salary and people with a salary <100k USD/year always tell you their salary, people between 100 to 200k tell you their salary 50% of the time and 50% of the time tell you ">100k", but people with salaries >200k always tell you >100k, you have a "missing not at random" time of censoring that is really problematic (unless you have other information that can make you distinguish these groups). My initial thought was that a split in time for the new scale would not be that kind of situation.