I have a set of observations that each consists of different levels. For example, I ask a $P$ individuals $N$ questions, each question with a possible $k_n$ discrete responses. This produces a table like this:

$ \begin{array}{|c|c|c|c|} \hline & \textrm{Person 1} & \textrm{Person 2} & \textrm{Person 3} \\ \hline \textrm{Question 1 (1=A, 2=B,..)} & 1 & 1 & 2\\ \hline \textrm{Question 2 (1=K, 2=L,...)} & 1 & \textrm{NA} & 5\\ \hline \textrm{Question 3 (1=X, 2=Y, ...} & 3 & 7 & 1 \\ \hline \end{array} $

In this representation, although each row shares numbers, they are associated with different answers. In other words, I am use numbers to represent the answers, even though I could substitute them with the words which would make each row distinct.

My issue is the following: I have missing data. Some questions are missing answers from different persons. Also, there are far more questions than there are people. So $N \gg P$. I want to use Multivariate Imputation by Chained Equations (MICE) to impute the missing values. My ultimate objective is to perform dimensionality reduction on the persons so I can cluster, etc.. the different people together.

The problem arises from the fact that in typical cases, a column represents a single feature, whereas here it is compact and this is no longer the case. One solution is to one-hot encode each question such that I end up with a binary matrix. My concern is once I do this, I will end up not with missing entries, but with missing blocks of data and the dimensionality will increase significantly (there can be up to 10 answers per question).

Another solution would be to transpose my data matrix, so that each column represents a separate feature. I believe the issue with that is when I try to impute the data, I am sampling the conditional probability distribution of a person's answer given answers to other questions. I believe this is incorrect and I should be sampling the conditional distribution of a person's answer given the answers of other people to the same question. I want to impute a person's response based on correlations between their and other persons' responses to other questions.

What is the best way of approaching this problem?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.