I have a dataset with 15 binary covariates and a continuous response variable bounded between 0 and 1. The binary variables represent correct or incorrect answers on a short test and the response variable is a measure of the same test takers performance on a related but more advanced and reliable test. I would like to select the best variables and weights to predict the score on the more advanced test. What would be the best way of doing this?

PS. I'm not a statistician but a computer scientist with only basic statistics and machine learning in my portfolio.

(Side note: One idea I had was to use some kind of logistic L1 or L2 regularized regression, however, glmnet does not seem to accept non-binary response variables when fitting a logistic model, which I guess is reasonable for normal use. The built-in glm function does accept a (0,1)-bounded response but does not perform regularization. If this approach seems reasonable, any tips on suitable packages or would I have to implement it myself? Other ideas I had was using "normal" regularized regression, or perhaps Principal Component Regression, however, I have tried both these and they give very different results and neither perform very well.)


Linear model fits here pretty well. However, the very important thing is that you should take account on questions cross-connections.

To put it simply, let you have just 2 questions. You can't usually fit well a simple model like this:

Y = A0 + A1*q1 + A2*q2 + e

where A0,A1,A2 - coefficients (from R), q1,q2 - answers given (binary - [0,1]), and e - error (also from R).

What often fits better is

Y = A0 + A1*q1 + A2*q2 + A3*q1*a2 + e

So, more generally, instead of just one line that should fit all combinations of answers, you get several lines that fit conditionally on answers given.

However, be cautious with this approach as your feature vector might grow substantially which in turn might affect your model accuracy (through excessive complexity).

  • $\begingroup$ Hmm, very interesting. In the extreme case I guess the feature vector would have the size of the power set of features, which would be unfeasible with 15 features, but maybe adding just the combinations of any two questions would be enough? Am I right in guessing that this approach would work best for average test takers but underestimate the strongest test takers and overestimate the weakest ones? $\endgroup$ Feb 16 '16 at 12:54
  • $\begingroup$ It depends on your model. In the simplest form above you get a mean regression. But you can also create a model for a median and a quantile regression if you need it. $\endgroup$
    – Roman Kh
    Feb 16 '16 at 13:15
  • $\begingroup$ In order to decide which question combinations to add to your model, you have to analyze your data deeply to understand cross-correlation and covariation of different questions and then iteratively try various models in order to find the best one. It is a model selection which might be quite tricky. $\endgroup$
    – Roman Kh
    Feb 16 '16 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.