Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
We need to estimate the probability of a person getting a question right for a given content area given his history of getting questions in the same content area right in the past. We would also presumably have records on how others have done on this question and in this content area.
Is there a good way or ways of doing this?
(This is sort of an education theory question, but I couldn't find a better place to post this question.)
If I understand your question correctly, you have a set of items (pass-fail) and you want to assess the probability of endorsing the $k$th item given its preceding responses? If that's the case, what is usually done in psychometrics for educational assessment is to rely on Item Response Model, like the Rasch Model. In short, you model the probability of endorsing an item as a function of item difficulty and person ability (the more proficient an individual is, the more likely his response to an easy item will be correct). This assumes that the content you are assessing is unidimensional, and that the items can be ordered by difficulty on that scale. A Guttman model is rarely applicable, so we may allow for some "imperfect" response patterns (e.g., 111011101110000, the 4th and 8th items were failed although the examinee reached the 11th item before giving up), but the sum score is a sufficient statistic for the Rasch Model. Under this approach, you need to have responses from other individuals on the same set of items. To get an idea, look at the LSAT data set and the way it is analysed in the ltm R package.
I have described a psychometrical model, not a purely probabilistic framework for estimating the probability of failing after the $k$th item, with all other items right (this would follow a geometric law).
Sounds like a classic data mining task: Y_i = whether person i gets the question right, X_i (vector) = set of past performance of person i.
Using a set of past data on n people (i=1,...,n), you can fit a predictive model of the sort Y = function of X.
A variety of models can be used to predict the performance of a new person on this question. If you are short of data, then logistic regression and discriminant analysis would be good choices. If you have data on lots of people, then you could try more data-driven methods such as classification trees, k-nearest neighbors, or neural nets.
