I am trying to calculate a pearson's correlation between scores on a test to determine test-retest reliability. However, there are only 4 items on the test, with a maximum score of 4 and minimum score of 0. Some students have taken the test several times and some students have taken the test only 2 times. I would like to find a way to impute values to make complete data sets so that if Student A has only taken it two times, but Student B has taken it 8 times, I can fill in 6 missing data points for Student A to make 8 complete data sets. I am doing this as part of a PHP script so I can't use R or SPSS to calculate the missing data. I need to know how I can manually impute the missing values.

I am thinking that I randomly choose a number between 0 and 4 for each of the missing data points, and I do that 3 - 4 times so instead of 8 complete sets, I get 32 complete sets (is that right?) and then do the correlation analysis on those data sets. Does that sound like it will work without giving me biased or overstating the correlation?

UPDATE: The scores are not connected to a specific time, i.e. all the students aren't taking the test on the same day. I can get a list of scores for each student, and then put those scores into columns, but there is no time-connection among scores in a single column.

There actually several tests that are looking at different things. Each test has at least 2 items, but most have 4 items. For each of the tests, I exclude students that only have 1 score, giving me a minimum of 2 complete data sets, but up to 1, 3, 7, and even 10 additional columns, of course with some values missing (because the students that only took that test 2 times will not have any additional scores). I am not sure how this affects the type of correlation calculation I should do.

UPDATE: I just calculated how many times each student took each test.

Test 1
2 attempts - 117
3 attempts - 26
4 attempts - 8
5 attempts - 3
6 attempts - 3
7 attempts - 1
8 attempts - 1
10 attempts - 2
12 attempts - 1

Test 2
2 attempts - 42
3 attempts - 7
4 attempts - 2
5 attempts - 3
6 attempts - 1
9 attempts - 1

Test 3
2 attempts - 54
3 attempts - 7
4 attempts - 1
6 attempts - 1

Test 4
2 attempts - 20
3 attempts - 1

Test 5
2 attempts - 58
3 attempts - 12
4 attempts - 3
5 attempts - 1

Test 6
2 attempts - 23
3 attempts - 1

Test 7
2 attempts - 36
3 attempts - 7
4 attempts - 1
6 attempts - 2
7 attempts - 1

The reason to impute missing values is to keep my sample sizes larger. Given the current sizes, am I okay doing correlations on just the 2 complete data sets?

  • $\begingroup$ Why did some students take the test more often than others? Because if this is due to some students failing the test at first, then the missing values are related to the outcome and you cannot simply impute them. $\endgroup$ Commented Feb 3, 2018 at 6:51
  • $\begingroup$ I don't have information about why students take the test the number of times they do. The test is taken online and I have no contact with the students. Some students might be using the test for practice (as a way to study the concepts) or their teacher might be using the test in their class combined with training. $\endgroup$
    – wuijin
    Commented Feb 3, 2018 at 7:20
  • 1
    $\begingroup$ I don't see that you need multiple imputation to answer your underlying question about test-retest reliability; it might even be inappropriate as @FransRodenburg suggests. A random-effects model would be able to take into account differences among individuals while allowing for different numbers of attempts among the individuals. You don't need to have the same number of attempts by each individual. $\endgroup$
    – EdM
    Commented Feb 3, 2018 at 14:26
  • $\begingroup$ Could you walk me through how to do the correlation using a random effects model? I need to write a PHP script that will do the calculations so I can't just put the data into SPSS and click a button. The only tutorials I can find online show how to do it with R or other stats programs. $\endgroup$
    – wuijin
    Commented Feb 6, 2018 at 15:59

1 Answer 1


The standard way of doing multiple imputation is to fit a suitable Bayesian model (usually with very vague or weakly informative priors) and then to draw the missing values from the posterior predictive distribution.

In your case that could mean some kind of random effects model (to take into account that responses from the same person are probably somewhat similar) with some kind of correlation structure (e.g. just compound symmetric = a standard random effect on the intercept, if the different test occasions are just random days with no special meaning too them - e.g. not test #1= 10 years ago, 2= #5 years ago, #3= 1 year ago, #4=6 months ago, ..., #8= 1 day ago, when you might want to impose more structure on the correlations). In terms of an actual likelihood, you could either use a normal approximation and accept non - integer values (including <0 and >4) or use an item response type off model (i.e. there is a latent continuous variable in the background that according to some thresholds to be estimated gets mapped to your observed discrete outcomes).

Realistically, you would need a good general purpose MCMC sampler (e.g. Stan) coded by someone else to implement this in an acceptable amount of time. Or you could fit such a model using maximum likelihood and assume that this gives you a decent approximation to an posterior (asymptotic normality of posterior). Even better if someone else has already implemented this type of Bayesian or frequentist model, or multiple imputation for it.

  • $\begingroup$ Thank you for the explanation, but I didn't understand it. I am not a statistician. Can you explain how I would find a Bayesian model that fits my data? Also, regarding the time, I can map each score to a specific date, but the time lapsed between scores is different for each student. All I can do is line up the scores into columns such that there are a minimum of 2 complete data sets (2 columns), and then after that I might have missing values, without any connection to the actual time the test was taken. $\endgroup$
    – wuijin
    Commented Feb 3, 2018 at 7:23
  • $\begingroup$ this is not easy stuff. You may want to enlist a statistician's help. $\endgroup$
    – Björn
    Commented Feb 3, 2018 at 7:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.