1
$\begingroup$

I am trying to build a model where I want to measure the accuracy with which supervisors can predict the outcome of test taker's scores. For example, supervisors rate test taker's subjectively before taking a test based on a short interview. After the completion of the test, test results will be compared with the prior assessment.

Would a simple regression model with the subjective test score as the independent variable and the subjective prior assessment score as the independent variable make sense? I see that there might be a problem with cross-correlation but I'm having trouble understanding how. How could an appropriate model look like?

Thanks in advance.

$\endgroup$
0
$\begingroup$

My suggestion on measuring accuracy, this 2020 article: The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation:

The Matthews correlation coefficient (MCC), instead, is a more reliable statistical rate which produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (true positives, false negatives, true negatives, and false positives), proportionally both to the size of positive elements and the size of negative elements in the dataset.

$\endgroup$
1
  • $\begingroup$ As I understand, the MCC is only applicable on binary classification. But in my scenario, I would have scores from 0 to 100% and prior assessment ratings on a scale from 1 to 10. Is this correct? $\endgroup$ – Romsch Jul 24 '20 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.