We have data from 600,000 users that describes whether they observe each of 80+ binary features. That is, our data are a 600,000 x 80 binary matrix with user-profile.

We know from inspection that some features are positively and negatively correlated. Some positive/negative features exclude others. Most users have less than 10 positive features in their profiles.

We want to retrieve the profile of new users by asking them the minimum set of questions from the 80+ potential ones given this previous data.

The idea is to give a small set of questions (5-10) to new users. Those should provide the maximum amount of information in order to "cut" the number of plausible subsequent questions. After a user has answered the first set of questions, we would like to ask a next set that, again, "cut" the number of subsequent questions faster. It seems reasonable to take into account positive and negative responses.

Could you please provide me with some hints to how to implement this model? We would like to have:

  • A way to measure the distribution of the expected number of questions given to each user.
  • Some way to tune the "number of initial questions" provided.
  • The model should be preprocessed in order to be able to react fast to user input.
  • If possible, visualize the relationship between questions.
  • If possible, be able to control the expected number of questions (I guess by discarding low correlations).
  • If possible, update the model incrementally using new respondents (not critical)

We plan to prototype in Python and then implement in .Net, but any other hint/code will be welcome.

  • 1
    $\begingroup$ Two-mode clustering might do the core task for you: identify subsets of questions which can distinguish homogeneous subsamples of users. $\endgroup$ – ttnphns Jan 5 '14 at 9:25
  • 1
    $\begingroup$ Although clustering techniques are something that can be used as part of the solution, the core problem in my question is to minimize (local minima might suffice) the potential number of questions to a new user. Once we cluster, How would you choose the first set of questions?, Once the user reponses this set, How would you choose the second set of questions?, At any given time, How would you decide no more questions are relevant? Thanks for your link. $\endgroup$ – jruizaranguren Jan 5 '14 at 12:13
  • 2
    $\begingroup$ If you need a hierarchical decision scheme you might use classification tree analysis (also random forest, based on it). $\endgroup$ – ttnphns Jan 5 '14 at 12:24
  • $\begingroup$ Do you assume each user has one profile and the total number of these is known? $\endgroup$ – vqv Jan 11 '14 at 15:25
  • 1
    $\begingroup$ Yes, each user has a profile and the ratio of #different profiles / #users is about 1/3. I think classification tree analysis is a promising approach that I am going to explore. $\endgroup$ – jruizaranguren Jan 11 '14 at 19:53

Sounds a lot like a computerized adaptive testing (CAT) application. This is just one small hint, not an attempt at a comprehensive solution, so I hope others will keep the answers coming.

I'm assuming that you're hoping to predict responses to the unasked questions from an optimally small subset of questions to such a degree of accuracy that there is effectively no need to actually ask the questions to which the answers can be predicted from previous responses. Specifically, I'm assuming a couple things about your original meaning:

  • "Some positive/negative features exclude others." = Some features can be used to predict the absence of others very accurately, maybe even without any error at all.

  • "In order to 'cut' the number of plausible subsequent questions" = The purpose is to reduce the number of follow-up questions that mostly provide information that is redundant with information collected by already-asked questions.

If I've misinterpreted these parts, my hint may be misleading; otherwise, I think I'm at least pointing in the right general direction. I don't know much more about CAT than this general purpose that it serves, so I expect you'd be better equipped than me to efficiently study it further.

One other idea concerns a slightly different approach, whereby you'd try to reduce the overall number of questions you care to ask at all of future users. You could begin to do this by analyzing the latent factor structure of your existing data using something like multidimensional item response theory (MIRT; see, for instance, Maydeu-Olivares, 2001; Osteen, 2010). If you find that a lot of your items provide information about the same underlying factors, this could help you understand your total pool of information in terms of a shorter list of broader factors. If you find that list (of the latent factors in your set of questions) contains enough of what you really want to know, you might choose to eliminate some questions that don't predict the latent factors very well and don't provide other important information. You might even consider retaining only one or two of the items that best predict each latent factor, depending on what you ultimately want to do with these data. This tangential idea of mine assumes that some of your questions are disposable. Also, disposing some questions would probably only simplify your problem somewhat, not really solve it.

Also, I think both CAT and MIRT would assume that your binary data are indicators of (an) underlying continuous dimension(s). If that's not the case, both ideas may be misleading, and you might want to say a little more about the nature of your data to help inform future answers (or edits to my own).

  • 2
    $\begingroup$ You interpreted my poorly explained problem perfectly. The objective is to retrieve all the information about the user with the minimum set of questions. CAT has a different purpose (maximize exam precision). I'm not sure it suits my needs, although I'm reviewing item response theory.In my case two users might answer positively a set of different questions and that is right and it just means they have orthogonal profiles. I'm not interested in scoring users, just in knowing them. $\endgroup$ – jruizaranguren Jan 2 '14 at 8:09
  • 2
    $\begingroup$ We have built, with the help of domain experts, a set of questions that contains independent information. They might be related but we need all of them in order to build the complete universe of user profiles. We will make some latent factors analysis in order to evaluate our selection but I consider it a related issue. $\endgroup$ – jruizaranguren Jan 2 '14 at 8:10
  • 2
    $\begingroup$ My data consist in facts about users economy. They are all binary so far. Some examples: owns shares, hired people for a company, owns a pension scheme, bought a house last year, received heritage, is retired, etc. If your are retired you did not hired people for a company and so on. Perhaps case-based recommenders are more related to this issue, but I still have not found something similar. $\endgroup$ – jruizaranguren Jan 2 '14 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.