I administer technical interviews for a programming position. During the process I generate a single number 0-12 for each applicant's likelihood to succeed within the company. Over the last year I have an ordered list of 50 or so such numbers.

I suspect that over time, my judgment has changed and I am grading people easier (or harder) now than I was a year ago. What statistical analysis would I preform to get more information on this hypothesis? Since this is largely for my own curiosity, I would ideally like a tool that generates further insight rather than a simple hypothesis rejected/not rejected.

I realize that I could do something like take the first and last thirds and do a t-value test to check the likelihood both come from the same population, but this seems intuitively like it would be the wrong approach as it would be discarding the information that contiguous numbers are very likely from the same population.

PS. Yes, I understand the limitations of generating a number for such a thing at all, and the general fallibility of "expert judgement". The entire reason for gathering these numbers to begin with is so that I can start using something other than gut feel for improving our process, but I am very conscious of the tendency to trust data like this too much.

Full disclosure, I originally posted this on the math stackexchange but I just found this one and it seems the better location for it.

  • 1
    $\begingroup$ There are many, many possibilities here. See discussion at stats.stackexchange.com/q/85686/12258 and probably elsewhere. Are your measurements at regular time intervals (probably not)? Have you plotted the data? $\endgroup$ – atiretoo Sep 20 '15 at 15:57
  • $\begingroup$ @atiretoo No, the measurements are not at regular intervals, although I would be ok with assuming that they are as the interviews have been mostly regular. I've done a simple histogram but nothing about the time series. $\endgroup$ – George Mauer Sep 20 '15 at 16:12
  • $\begingroup$ @atiretoo to be honest, that post is a bit technical for me. It's been a long time since I was a stats tutor in college. I was hoping for something like a name of a particular technique that I could read up on, plug into r, and go. $\endgroup$ – George Mauer Sep 21 '15 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.