How to measure student skill level based on a set of questions answered at different points in time and of different difficulties levels?

I am working on an e-learning system with a friend for our final year (Computer Science) project which is part of the under-graduate programs mandatory 'courses'.

I have a question about making inferences and therefore gauging the skill level of the 'subject (student) at hand'

We have gathered the following data thus far for every attempt made at answering a question:

1. Unique ID of the question
2. Attempt number
3. Outcome of the attempt (Correct/Incorrect)
4. Difficulty of the question (assigned a nominal score between 1-5)
5. Date and Time at which the attempt was made

How can I use the entire data set regardless of what point in time the answers were given and judge the skill level of the student from that particular model?

I know I could just look at the last X number of attempts for a section of a particular difficulty level and pick up a trend from that, but is there a better way to do this?

• I'm struggling to understand your question. This is how I understand it: A student answers a series of questions you created. He/she can take multiple attempts to get the answer correct. Then, you wish to take the data and determine the student's skill level from 1) how many times they attempted it and 2) how difficult the question was. Is this correct? Will the student have infinite attempts so they always end up getting the correct answer eventually? – Hotaka Sep 19 '13 at 20:04
• That's pretty much the gist of it. Since each section has multiple questions I would like to take data for an entire set of questions answered and use the information you mentioned (number of times the question was answered, and how difficult the question was) to figure out the users skill level. – SPI Sep 19 '13 at 20:37
• Ok. I can think of a whole bunch of ways to do this. How did you figure out the difficulty? If you determined difficulty subjectively, that was probably not a good idea. Are the students told to try to get the questions right in the fewest number of attempts? Are these multiple choice questions? – Hotaka Sep 19 '13 at 22:25
• The difficulty level is judged in two parts: 1) When the question is first inserted into the system the sum is broken down into the number of operators involved as well as the size of each operand. From this data we get a rough estimate of the questions difficulty 2) We adjust the difficulty of each question based on the answers received. We take into account the average number of attempts to get the answer right as well as the average skill level of the users who got them right to adjust the difficulty of the question. – SPI Sep 20 '13 at 15:23
• We haven't decided on the mode of questions yet, we are doing some research on the pros and cons of the Multiple Choice Questions and the regular kind in terms of how they aid the user in learning new concepts. We also don't explicitly ask the users to get the answers right in the fewest possible attempts, but that should be a given since they are trying to solve a question. – SPI Sep 20 '13 at 15:34

The easiest way to determine skill level would be to simply add up the number of questions correct for each student on the very first attempt. Unless you have a clear reason why attempt count is important, it is vague whether number of attempts would represent skill level. Plus, item difficulty is taken into account even if you disregard it; students who get the difficult questions right will simply get more points. This is the way most tests are assessed.

You should try calculating the reliability too. look up kuder-richardson 20. The idea is that students who get high overall scores should be the ones that are more likely to answer the difficult questions (ones that not many get correct) correct, and vice versa. If the reliability is low, it means your test was poor at discriminating students who have good/bad knowledge, which is the whole point of the test.

• Excellent suggestions, Hotaka. I would love to upvote your post but I don't have enough reputation points. I have noted your suggestions, the Kuder-Richardson formula should come in handy. I will read into this some more. I am glad I came here. I will eventually accept your answer, but I am hoping for a little more feedback and I hope the incentive of "correct answer" will drive more people here ;) – SPI Sep 20 '13 at 15:28

There are quite a number of elaborated test theories for estimating person paramters based on test data and soemtimes simultaneouls estiamting item difficulties. I wouild look at Item Response Theory and Rasch Scaling procedures. They might not cover your particular concern about time points, though. I would argue that no statistical model will be able to capture how much learnign is going on between time points, unless you build a psychological model of learning trajectories that is precise enough to make valid projections. If you assume hetereogeneity in learnign progress, you might not have enough data to assess learning rate and skill level simulatenously while still controlling for error rates....

But I would first commit to a test theory, and then define the measurement problem and repeated-testing and learning problem.

I would like to add to @jank's answer. I also think that item response theory is the way to go. If you haven't heard of it, yet, there might be a bit of a learning curve. But if you are really interested in test theory, you should definitely have a look at it, as it entails a lot of the advances that have been made in this area. A good and detailed book, that is freely available is that by Reckase (2009).

The article about the eRm-package for R also has a nice introduction. What's more important, there is a section about using IRT for longitudinal data, which is of relevance to you.

References:

Reckase, M. (2009). Multidimensional item response theory. Springer.
Mair, P., & Hatzinger, R. (2007). Extended Rasch modeling: The eRm package for the application of IRT models in R. Journal of Statistical Software.