I have a set of problems to analyze from a programming contest. My work is to rate the question according to its difficulty (easy,medium,hard) by analyzing the submission rate,accepted solutions rate and failed solutions rate of the participants. Do I need any more parameters to analyze the difficulty and which algorithm should I use?

Edit #1:

Rare Case

What if a problem with highest difficulty has no submissions at all? How can this case be evaluated?

Edit #2:

About the contest

It is an online competitive programming contest like ACM ICPC. Questions are available to all the participants (all questions). They can start from any question. There will be 10-12 questions max. There is no submission limit for the questions. You can submit until your answer gets accepted.

  • $\begingroup$ Can you tell us more about the contest? Do all participant see all the problems? Do we have the number of submissions per question per participant? $\endgroup$ – Cam.Davidson.Pilon Feb 20 '17 at 5:33
  • $\begingroup$ @Cam.Davidson.Pilon Ok I will include those in my question. $\endgroup$ – PSN Feb 20 '17 at 12:54
  • $\begingroup$ Do you only have the rates for each question or do you have the data for each contestant? If you have the data for each contestant such whether or not he scored on a question, then we could rate the difficulty such as the average total score of those whose solution were accepted. $\endgroup$ – Nikolas Rieble Feb 28 '17 at 16:34
  • 1
    $\begingroup$ Please illustrate your data with a toy example. $\endgroup$ – Nikolas Rieble Feb 28 '17 at 16:35
  • $\begingroup$ Are the questions presented in a random order or always in the same order? $\endgroup$ – jbowman Mar 2 '17 at 20:57

If you have not any label for the problem, you face an unsupervised learning problem. You can use clustering methods such as k-means (with k=3) to group the questions into three groups, where the problems in each group are similar to each other and dissimilar to questions in other groups. Each question is represented by a vector containing the number of submissions, accepted, and failed. After clustering, you can determine the difficulty level of the questions in each group, by investigating a small number of questions in each group.

  • $\begingroup$ But there is still a problem. Consider a question with highest difficulty level and there are no submissions for that. How can this method rate such a problem? $\endgroup$ – PSN Feb 20 '17 at 5:12
  • $\begingroup$ No submission is a clue of the difficulty of the problem. When you cluster your problems, the problems with a low number of submissions are probably placed in the same clusters. Maybe you should consider more than three clusters and then, merge some clusters by hand. $\endgroup$ – Hossein Feb 20 '17 at 6:27
  • $\begingroup$ You could also add a dimension for "non-response". I'd probably go with "non-response", "accepted" and "failed" as three dimensions instead. $\endgroup$ – jbowman Mar 5 '17 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.