This question was asked in an interview

At a company, say ABC, the hiring department collects data on interviews conducted by its employees. The data we have contains 4 variables, timestamp, employee_id, candidate_id and a candidate_score that ranges from 1 - 4 where each record represents an interview. We do not know if the candidate was hired or not.

Consider that each employee interviews 30 candidates and each candidate is interviewed by 6 employees.

The question: ABC wants to figure out what is wrong with the hiring process using this data. What kind of statistical analysis would you use?

Any ideas on how to approach this problem would be helpful.

  • $\begingroup$ Add the self-study tag. $\endgroup$ – Michael R. Chernick Apr 1 '18 at 22:40
  • 2
    $\begingroup$ This question seems vacuous without an actual explanation of what anomalies have been observed during the hiring process. $\endgroup$ – Alex R. Apr 20 '18 at 23:59
  • $\begingroup$ The problem with ABC's hiring process is asking stupid questions like this in interviews. $\endgroup$ – Mark L. Stone Apr 21 '18 at 14:32

As Alex R. pointed out, the question may be vacuous if you do not explain what defines anomalies. I will assume that the question to answer here is to explain the target to track its potential deviations, based on your variables, and eventually identify root causes. For the sake of simplicity, I will refer to candidate_score as target, and other parameters as variables. Since the question is designed for interviewing purposes, I assume that by no mean, answers from candidates should be exhaustive.

Analyzing and visualizing variations per variable

Since you have very few variables, first thing to do would be to analyze and visualize potential score variations based on one variable.

  • timestamp: Calculate average score per period (week day, month, year), depending on the exhaustivity of your dataset. Start simple, look if you see any evolution across time (plotting time-series or histograms), or decompose the series if you are given a sufficient time-window into a trend/seasonal/remainder pattern. This would highlight seasonal phenomenon, e.g. potential keener notes on Friday than on Monday, or during holidays period.
  • employee_id: Given the number of candidates interviewed per employee, it may be relevant to calculate the average score given per employee and examine the associate distribution through a boxplot. You will be able to track deviations in the way employees score candidates. A tight boxplot implies that they grade candidates in a uniform standard way (presumably according to rules clearly defined for the recruiting process), while a dispersed boxplot means that scores are subject to consistent variations depending on the employee. Additionally, if you have very few employees, you can directly calculate the average awarded score per employee and identify which ones are keener/harsher.
  • candidate_id: This one would be trickier to evaluate, as candidates are supposed to be independent from the process. However, you can calculate the standard deviation of their score to see if they are perceived by employees in a uniform way (or not) / or that the grading rules make sense in the process. Again, you can boxplot, and examine the associated distribution, such as for average score per employee_id.

Examining interdependency

You may be interested in examining dependencies between your variables, and their influence over the target. You can combine your variables and perform the following analysis.

  • Transform your timestamp into a categorial variable (e.g. month or weekday) and calculate the average score per employee per period. This would be close to performing a correspondance analysis. However, the pertinence of such analysis would be limited as you have very few grades per employee : 30 grades divided by 5 or 12 would return statistically insignificant results.
  • Dummify your variables and target, and look for correlation. This may highlight the fact that some employees are dedicated to specific period of time, but you would not gain much from what has been performed before.

Obviously, the analysis and reliability would clearly depend on the number of employees/candidates that were involved in the process, as well as the period covered by your dataset. Nonetheless, it can be a starting point, especially given the fact that in an interview, you will have limited time to develop your ideas.

  • $\begingroup$ (+1) Rather than, or as well as, looking at the average score given by each employee, wouldn't it be better to look at the deviation of that score from the average score given to the same candidates by other employees? After all, the mechanism by which employees are allocated to candidates is not explained; & it's easy to imagine that employees may be selected to interview according to the job the candidate's applying for, & that some jobs attract better applicants than others. $\endgroup$ – Scortchi - Reinstate Monica Apr 21 '18 at 11:36
  • $\begingroup$ This is why I suggested to calculate the standard deviation per candidate_id, as it measures the score deviation regardless of the average of the candidate. Of course, it is possible to replace standard deviation by absolute average deviation.But agreed, the process should also include additional information such as a job_id to which candidates applied. $\endgroup$ – AshOfFire Apr 21 '18 at 14:38
  • $\begingroup$ Quite so: my point's that it's worth looking to see how much of that variation in the scores given to each candidate is explicable by a general tendency for some interviewers to give higher or lower scores. (It might not be a concern if different interviewers have the task of assessing different aspects of the candidates' suitability.) $\endgroup$ – Scortchi - Reinstate Monica Apr 21 '18 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.