We have data from 600,000 users that describes whether they observe each of 80+ binary features. That is, our data are a 600,000 x 80 binary matrix with user-profile.
We know from inspection that some features are positively and negatively correlated. Some positive/negative features exclude others. Most users have less than 10 positive features in their profiles.
We want to retrieve the profile of new users by asking them the minimum set of questions from the 80+ potential ones given this previous data.
The idea is to give a small set of questions (5-10) to new users. Those should provide the maximum amount of information in order to "cut" the number of plausible subsequent questions. After a user has answered the first set of questions, we would like to ask a next set that, again, "cut" the number of subsequent questions faster. It seems reasonable to take into account positive and negative responses.
Could you please provide me with some hints to how to implement this model? We would like to have:
- A way to measure the distribution of the expected number of questions given to each user.
- Some way to tune the "number of initial questions" provided.
- The model should be preprocessed in order to be able to react fast to user input.
- If possible, visualize the relationship between questions.
- If possible, be able to control the expected number of questions (I guess by discarding low correlations).
- If possible, update the model incrementally using new respondents (not critical)
We plan to prototype in Python and then implement in .Net, but any other hint/code will be welcome.