7
$\begingroup$

I have a number of students and their scores across a series of tests, all of which have equal importance in my model. I would like to identify the top scoring students (e.g. top 10%).

My first idea was to average percentile ranks and work from there. However, I couldn't come up with a way how to attribute statistical significance to the results I'd get.

My second idea is the following. First, take one test at a time and convert the scores of all students on this test to $z$-scores. Second, run a one-sample $t$-test for each student, with the following hypotheses:

$\mathcal{H}_0$: Average $z$-score across all tests is equal to 0.
$\mathcal{H}_A$: Average $z$-score across all tests is larger than 0.

This procedure would give me a $p$-value for each student. I could then set a threshold $T$ such that only approximately 10% of students achieve $p$-values lower than $T$.

Does this approach make sense? What could be its caveats? On the other hand, if it doesn't make sense, can you sketch an alternative?

My interest is in an approach that deals with the class of problems in which I have a set of objects (in the order of thousands) and their scores across a series of equally-important events, i.e., not necessarily only the student-test scenario.

$\endgroup$
  • $\begingroup$ What do you mean by 'theoretically correct' in that context? $\endgroup$ – Glen_b Jun 21 '14 at 17:29
  • 1
    $\begingroup$ I've changed the wording. $\endgroup$ – John Manak Jun 21 '14 at 19:24
  • 4
    $\begingroup$ To recommend a particular analysis I'd want to understand a lot more about what you're seeking - what properties you want the calculation to have, for example. My first instinct would be to consider your statement "identify the students who consistently achieved high scores (e.g. top 10%) across the events" and try to actually do that. So "has the greatest frequency of being in the top 10%" rather than has the highest average ranking. So it would come down to a discussion of whether the things you suggested are closer to what you seek, or your 2nd sentence is - or something else again. $\endgroup$ – Glen_b Jun 22 '14 at 1:56
  • 1
    $\begingroup$ Have you considered using Item Response Theory methods? $\endgroup$ – Tim Nov 24 '14 at 12:43
  • 2
    $\begingroup$ Scores from simple marking tend to correlate closely to irt scoring procedures (~.95) so Item response theory would probably only be worth the extra effort if you wanted to refine the scales. It doesn't sound like the op is as interested in this. $\endgroup$ – Sue Doh Nimh Nov 25 '14 at 11:14
3
+50
$\begingroup$

I agree with the comments above that this would be easier if the question were a little more precise.

Here are some things to think about.

set.seed(1)

## simulate data: refine to reflect attributes of interest

n <- 30  ## test count
m <- 100 ## student count

## student "ability": student rows, test columns
sa <- matrix(round(runif(m*n,30,80)), nrow=m)

te <- 5 ## test error

## test scores:
ts <- matrix(round(rnorm(m*n, sa, te)),nrow=m)
rownames(ts) <- seq(m)

Method A:

Use a summary measure that collapses student variability across tests.

## test score summary
tss <- apply(ts,1,mean)

qqnorm(tss); qqline(tss, col=2)

enter image description here

Taking tss to be a sample from the normal distribution with matching parameters:

ps <-pnorm(tss, mean(tss), sd(tss))

## top 10% students
(a <- rownames(ts[ps>0.9,]))

[1] "4"  "14" "23" "30" "39" "43" "45" "49" "50" "68" "69" "80"

Method B:

Take z-scores within tests, report p-values from one-sided t-tests within students.

z.ps <- function(ts) {
 zs <- scale(ts) 
 st <- apply(zs, 1, function(s) t.test(s, alternative='greater'))
 unlist(lapply(st, function(s) s$p.value))
}

ps <- z.ps(ts)

## top 10% students
(b <- rownames(ts[ps<0.1,]))

[1] "3"  "4"  "8"  "14" "23" "30" "39" "43" "45" "49" "50" "68" "69" "80" "83"
[16] "96"

This results in a more "permissive" threshold. It does try to account for variability across tests within students but the normal assumption is sometimes questionable, even for larger n.

See here (graph not shown), and repeat methods with differing m,n:

op <- par(mfrow=c(3,4))
for (i in sample(n,12)) { qqnorm(ts[i,]); qqline(ts[i,],col=2) }
par(op)

This is in part because of the way these data are simulated. It is perhaps not very realistic to model "student" abilities as uniform on an interval and uncorrelated within human subjects.

The point is not the defence, or otherwise, of the naivety of the simulation. But rather to reinforce that if your "students" are performance metrics from arbitrary processes, the attributes of these will influence what constitutes a sensible approach.

Hope that helps.

$\endgroup$
3
$\begingroup$

The method that you describe seems quite convoluted (and perhaps fairly inaccurate). Presumably, the tests measure some sort of trait (e.g., intelligence), or ability (e.g., math ability), or experience (e.g., learning) and you want the top 10% of students who score highest on this construct. You also mention having a series of tests, which suggests you have several scores. If these tests are related, and to you they measure the same ability, then why do not you not simply sum up the scores for all tests? You could then just use the top 10% on these raw scores to identify the top 10%.

If you want to go a bit more complex, you could use exploratory/confirmatory factor analyses to compute weighted scores based on the associations between each of the scales and the ability you wish to assess.

You could also use aspects of item response theory to identify the strongest test takers.

One issue is your mention of "statistical significance". I don't know what you are referring to here. It seems like you want to identify individuals, and this seems more descriptive than relevant to statistical significance and null-hypothesis testing. If you want some sort of index of how reliable your estimates are, then estimates or error/information will be valuable, not statistical significance per se.

$\endgroup$
1
$\begingroup$

Personally, I don't like the premise. You asked and I will work with it, but I do not like the fundamental here.

Personal aside on teaching

I personally think that a teacher has 2 jobs, and neither is improved by ambiguity:

  1. to teach and certify the student has all the fundamentals down. If something isn't a fundamental, then they don't need it. If they need it, then it is a fundamental. Every student passing a class should have 100% of the fundamentals down. No student passing a class should be lacking in any of the fundamentals - if they lack they should fail the class.
  2. to teach the student how to teach themselves. By this I mean that all relevant learning methods are part of the fundamentals. The student who has this is capable of building in all relevant ways upon his fundamentals to arbitrary levels of exceptionalism. No student passing a class should be unable to teach himself in any relevant way. Every student passing a class should be able to teach themselves in each and all of the relevant ways for that subject.

How I win at "Pick-em"

I play a version of fantasy football called "pick-em". My results are at least as good as the Las Vegas "spread" values.

The problem can be stated as:

  • given 16 teams, and some recent history of how they play (I only need points per game)
  • given two of the 16 teams who will content in the next game
  • determine which is most likely to win
  • determine a scale indicating how much more likely to win one is than the other

My approach:

  • Set the metric: For all past games the team played, if they won indicate point differential as positive, otherwise indicate it as negative
  • Establish the coordinate system: For the game of interest, put one team at coordinate x=1, and the other at x=-1. Each of the differential scores becomes one y value. That mean at x=1 there will be as many points as games that are being considered relevant history. If the last 5 games are considered relevant history then there are 5 points at x=1 indicating relevant scoring of team x=1 and there are 5 points at x=-1 indicating relevant scoring of team x=-1.
  • use the Theil-Sen estimator (or its relatives) to determine the median slope between all pairs of points. If the slope is positive then the team at x=1 will win, otherwise the team at x=-1 will win. The slope is a measure of how likely a team is to win. If the slope is shallow the win-loss is uncertain, but if the slope is high then the win-loss is more reliable.

I should disclaim that all standard disclaimers apply so nobody writes me hate mail saying they lost money betting using this method. If you bet then it is your decision, not mine.

Applying it to kids

You can make a matrix (aka graph) comparing all kids. at the cell (i,j) which compares kid "i" versus kid "j" put the slope of the T-S estimator. If kid "i" typically wins, the slope is positive, otherwise it is negative.

Make row sums. That is going to give you a 1-axis robust estimate of performance. A higher value for the row sum means that the student overall is higher performing. A lower value for the row sum means that the student overall is lower performing. It will be the best overall estimator of comparative performance. It will tell you nearly nothing about absolute performance. It will tell you nothing about meeting the criteria that I gave. It will, however, be an excellent estimator of comparative performance. It will allow clustering.

There is a textbook "reality check" for univariate data called the 4-plot. (link) If you want any sense of what is going on then use this. It is going to show you trends, outliers, something about the dependence of the grade of kid "i" versus "i+1", and the nature of the distribution. There is hand-waving saying in the limit of infinite samples everything is gaussian. In reality nothing is gaussian, not even pseudorandom numbers meant to look gaussian. The true distribution can tell you about clusters, clumps, (probability modes), and outliers.

Best of luck

PS: If you want example and source code for textbook TS (not my variant) then I will provide it. Please request.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.