A problem I've been toying around with:
Company A helps a group of students to prepare for a standardized test. A perfect score on the test is 100, but most of Company A's students receive scores between 60-90. To help their students prepare to take the real standardized test, Company A offers a series of 10 practice tests on consecutive weekends.
Company A would like to assess whether any of the tests in their practice test line (some of which have been created by the company itself) are, on average, too hard or too easy. Company A has a data set consisting of hundreds of students, all of whom have taken the full series and who eventually take the real test. Since Company A helps students prepare for the test, they expect that, given tests of equal difficulty, students' ability will improve over the 10 tests. Company A does not wish to assume that the improvement is necessarily linear.
Here are my questions:
What sampling procedure should the company use? Company A wants to sample as few students as possible because accessing old data is time intensive.
What statistical tests should the company apply to the data to see whether there is a statistically significant difference in difficulty between tests?
How should Company A account for the fact that they expect students' scores to improve naturally over the course of the 10 tests?
(This is a slightly concealed version of a problem I ran into at work, and it seemed like a tricky statistical problem. But maybe I'm just not familiar with relevant tools.)