1
$\begingroup$

I have user responses to different types of surveys (e.g. "car survey", "lifestyle survey",...). Most of the users have answered just a very small number of surveys. I would like to predict the response on the remaining unanswered surveys based on all the data available (answered surveys and user demographics). The surveys have categorical variables and numerical variables as well as single and multiple responses.

The approach I have begin with is to treat the problem as a recommender system, where the items are the different questions. However I am not sure this is the best possible approach. Specifically I am having troubles dealing with categorical variables and the multiresponse questions as most of the literature about recommender systems focus on the rating user-item problem.

I am also experimenting creating a glm model for each question. It gives me good results but it is slow and it does not scale very well. I have ~150k users and 200 questions.

Do you know any better approach to the problem? How can I manage the problems with the recommender system approach (categorical variables, multiple response)?

Thank you!

Improve this question
$\endgroup$
2
$\begingroup$

I would see this as a missing data problem. One very common approach for such problem in surveys is imputation. In your case, since you have multiple variables to impute, but also multiple fully observed variables, you could try using a Multivariate Imputation by Chained Equations (MICE) approach, which is implemented in R in the mice package. A similar approach is the Sequential Regression Imputation Method (SRIM), which is implemented in IVEware available both in a stand-alone and SAS version.

The MICE implementation is very convenient because it "automatically" identifies the types of variables being imputed and it selects an appropriate model for the imputation. In the SRIM implementation you have to specify the types of each variable you are imputing, but it is more flexible, in the sense that you can use a variety of constrains, such as restrictions and bounds for the imputed values.

However, both approaches may take a long time to run, especially given the size of your data.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you very much for the response! I have tried mice and it seems to be focused on imputing plausible missing values for doing something later with them. However, we are looking for each individual and each question, to make the best possible prediction of its value, we are concerned in a per user treatment. $\endgroup$ – kahlo Dec 11 '14 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.