Statistical Test for Analyzing Practice Test Data

Question

A problem I've been toying around with:

Company A helps a group of students to prepare for a standardized test. A perfect score on the test is 100, but most of Company A's students receive scores between 60-90. To help their students prepare to take the real standardized test, Company A offers a series of 10 practice tests on consecutive weekends.

Company A would like to assess whether any of the tests in their practice test line (some of which have been created by the company itself) are, on average, too hard or too easy. Company A has a data set consisting of hundreds of students, all of whom have taken the full series and who eventually take the real test. Since Company A helps students prepare for the test, they expect that, given tests of equal difficulty, students' ability will improve over the 10 tests. Company A does not wish to assume that the improvement is necessarily linear.

Here are my questions:

1. What sampling procedure should the company use? Company A wants to sample as few students as possible because accessing old data is time intensive.

2. What statistical tests should the company apply to the data to see whether there is a statistically significant difference in difficulty between tests?

3. How should Company A account for the fact that they expect students' scores to improve naturally over the course of the 10 tests?

(This is a slightly concealed version of a problem I ran into at work, and it seemed like a tricky statistical problem. But maybe I'm just not familiar with relevant tools.)

Answer 1

1) Do power analysis to find out how many students are needed to find a desired effect size.

2) Do realise you are doing 10 different tests, so you have to account for inflated type I error (False Discovery Date), and a simple, say alpha of 0.05 per test is not a good choice here.

Question for point 1 and 2 to ask yourself: is it more important to find a test if it works (power), or is it more important to prevent from getting false positives? You should probably find a right balance.

3) You could do simple t-tests between (t being time). t1 and t2, t2 and t3 and so on.

Problem is that, if I understand the situation correctly, effects found could also originate from the test before the test measured. That is: when testing whether t4 is better than "baseline" t3, it might be there is an effect of t1 or t2 causing t4 to be higher.

Perhaps a nice and easy way to find out which test seemed to have helped, is to graph all test scores together. That is, on the X-axis the different tests (test1,test2,test3), and on the Y-axis the average scores. You should see an upwards trend (like you predict), but in the mean time can see if there might be tests that perform better than expected. If you add a linear regression line, you can then easily see which scores are above it. You can also see if there is some "non-linear" relationship in the sense that the first 3 tests could be great, next 3 bad, while the last are better again.

Answer 2

I would examine the practice tests using a Rasch analysis method, to determine how the items in each of the 10 practice tests line up across the range of difficulty. To do this, I would take the scores of around 30-50 students for each test, ensuring that this was the first test that the students had undertaken. I would then examine the item scores and student scores, separately for each practice test, to determine how the items are performing on each test. A Rasch analysis will, for example, show you whether the items are generally too easy or too hard.

To perform this analysis, you will need to ensure that your subset of subjects incorporates a range of abilities that the test is measuring: failure to ensure that the test scores are based on a range of abilities is much worse for this method than not having a large sample size.

You'll need particular software to do this. There are various options on the market. This answers your questions 1 and 2.

With respect to question 3, you could do a repeated measures test of the subject scores, once you have them scored using Rasch analysis. You will need to counterbalance test presentation order so that test ordering does not confound your results. A simple comparison of overall test scores should be sufficient. But you won't be able to do this accurately until you have corrected the practice tests for difficulty, as you need alternative forms of the practice test, and you may not have that yet. Depending on how different the practice tests are, counterbalancing may be insufficient to overcome this effect.

Answer 3

What sampling procedure should the company use? Company A wants to sample as few students as possible because accessing old data is time intensive.

You'll need to do a power analysis to determine the number of students that need to be sampled to adequately assess statistical significance.

What statistical tests should the company apply to the data to see whether there is a statistically significant difference in difficulty between tests?

You can set up this whole analysis as a repeated measure ANOVA. Testing for an interaction effect between student test exposure (one of your factors will be test exposure number), should address this.

How should Company A account for the fact that they expect students' scores to improve naturally over the course of the 10 tests?

I'm unclear whether this concern is in relation to [2], or to another statistical comparison of interest. Taking a repeated measures approach will account for this in the model.