5
$\begingroup$

Here is the link in question: http://applymagicsauce.com/documentation.html

When the Cambridge University Psychometric Center's "Apply Magic Sauce" defines how their Prediction Accuracy (AUC) system works, this is what they say:

Prediction accuracy is expressed as the correlation between the AMS prediction and the actual score. Accuracy of 1 indicates a perfect accuracy, whereas the accuracy of 0 indicates a random guess.

What is "the actual score"? In simple english, how is it calculated? I'm relatively new to these concepts, and I would just like to know how accuracy rates are calculated in machine learning.

When the documentation page states:

The model was build (and the accuracy validated) using a sample of 98,000 people.

What exactly do they mean? How was it validated, and what is the simplest way of doing so with a text classification system?

Improve this question
$\endgroup$
4
  • $\begingroup$ Without reading the link, AUC typically refers to area under the ROC curve. However, that does not fit the description, since a random model has area under the ROC curve equal to 50% rather than 0. Secondly, the ambiguous use of accuracy in the quoted description is very confusing. $\endgroup$
    – Marc Claesen
    Commented Mar 23, 2015 at 21:36
  • $\begingroup$ What is ROC? @MarcClaesen $\endgroup$
    – seanlevan
    Commented Mar 23, 2015 at 21:38
  • $\begingroup$ Receiver Operating Characteristic. $\endgroup$
    – Marc Claesen
    Commented Mar 23, 2015 at 21:43
  • $\begingroup$ But it seems to be measuring the ratios of false positives to true positives- how does it really know if the algorithm has interpreted something incorrectly? Wouldn't only a human be able to determine that? I'm new to this, as you can see. @MarcClaesen $\endgroup$
    – seanlevan
    Commented Mar 23, 2015 at 21:54

1 Answer 1

Reset to default
5
$\begingroup$

On the page linked, there are two metrics used to measure predictive accuracy of the algorithm.

1.The first is the usual Area Under the Curve (AUC) of the Receiver Operator Characteristic. As explained in the comments, AUC ranges from 0.5 to 1, with 1 being perfect classification and 0.5 being no better than luck.

enter image description here

Since there are only two outcomes, the algorithm can either classify positive correctly, or incorrectly. Similar to flipping a coin, chance will predict correctly 50% of the time, giving 0.5 AUC a "worthless" accuracy. As you increase from 0.5 to 1, your algorithm gets better and better at correctly classifying outcomes.

These algorithms are usually trained on a training set to tune parameters and tested on a test set to determine their accuracy and applicability. The AUC is found on the second table on the page.

2.The second of the metrics used is simply the correlation between the predicted outcome and the actual outcome. This was used because the outcomes were not binary; they were linear, and thus classification was inappropriate.

This has the range 0 to 1, as there can be 0 correlation between the predicted and actual outcomes. The correlation is found on the first "linear" table on the page.

As for the second part of your question, the 98,000 person sample that they mentioned most probably had known outcomes. For example, if your algorithm predicted a case (1), you could compare it to the actual known data to see if your algorithm predicted a true positive (TP) or a false positive (FP). This is how they measure the accuracy.

As for the text classification system, you would need to provide more details, perhaps a separate question for that might be appropriate.

Improve this answer
$\endgroup$
19
  • $\begingroup$ So that sample of "known outcomes" was most likely croudsourced? So it's basically a probability? Thanks again for your response. $\endgroup$
    – seanlevan
    Commented Mar 24, 2015 at 0:38
  • $\begingroup$ @seanlevan Perhaps not crowdsourced, but collected in some way by the researchers. May be a survey or a questionnaire of some sort, dependant on the outcome. Those sample sizes are huge for psychometrics, so it might be an online thing. $\endgroup$
    – Chris C
    Commented Mar 24, 2015 at 0:42
  • $\begingroup$ Thank you for your explanation, and the way that you simplified it to make sense. AUC ROC is basically just a plot comparison of the two variables, false positives and true positives? $\endgroup$
    – seanlevan
    Commented Mar 24, 2015 at 0:45
  • $\begingroup$ @seanlevan The ROC is a graphical comparison between TP and FP, while the AUC is the area under that curve. Just wanted to make that distinction. If you're ever looking to see if one model significantly classifies the same data better than another, look into Delong's Test or bootstrapping. Just if you're ever doing it yourself. Glad to be of help. $\endgroup$
    – Chris C
    Commented Mar 24, 2015 at 0:49
  • $\begingroup$ So the image in your post is not AUC? What's the difference then? I don't think I absolutely understand the distinction. $\endgroup$
    – seanlevan
    Commented Mar 24, 2015 at 1:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.