6
$\begingroup$

I am using scikit-learn in Python and they define a quantity called score. It's defined in the middle of the documentation page.

Reproduced here:

Returns the coefficient of determination R^2 of the prediction. The coefficient R^2 is defined as (1 - u/v), where u is the regression sum of squares ((y_true - y_pred) ** 2).sum() and v is the residual sum of squares ((y_true - y_true.mean()) ** 2).sum(). Best possible score is 1.0, lower values are worse.

A few questions regarding this:

  1. What's the intuition behind this metric?

  2. What is considered a good score? What is considered bad?

$\endgroup$
2
$\begingroup$

Actually there are two different measures that are called correlations. Let us then call them little $r$, which is the Pearson correlation coefficient, and big $R$, which is what you have; a correlation (usually as $R^2$) adjusted for a generalized residual. Now $|r|=|R|$ only when we restrict ourselves to ordinary least squares linear regression in $Y$. If for example, we restrict our linear regression to slope only and set the intercept to zero, we would then use $R$, not $r$. Little $r$ is still the same, it just won't describe the correlation between the new regression line and the data anymore.

Little r is normalized covariance, i.e., $ r= \frac{\operatorname{cov}(X,Y)}{\sigma_X \sigma_Y}=\frac{\sum ^n _{i=1}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum ^n _{i=1}(x_i - \bar{x})^2} \sqrt{\sum ^n _{i=1}(y_i - \bar{y})^2}}$. Finally, $r^2$ is called the coefficient of determination only for the linear case.

Big $R$ is usually explained using ANOVA intermediary quantities:

  • The total sum of squares proportional to the variance of the data: $\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }$ $SS_\text{tot}=\sum_i (y_i-\bar{y})^2,$

  • The regression sum of squares, also called the explained sum of squares: $SS_\text{reg}=\sum_i (f_i -\bar{y})^2,$

  • The sum of squares of residuals, also called the residual sum of squares: $SS_\text{res}=\sum_i (y_i - f_i)^2=\sum_i e_i^2\,$

The most general definition of the coefficient of determination is

$R^2 \equiv 1 - {SS_{\rm res}\over SS_{\rm tot}}.\,$

Now, what is the meaning of this $r^2$ or more generally $R^2$? $R^2$ is the explained fraction and $1-R^2$ is the unexplained fraction of the total variance.

What is a good coefficient and what is bad one? That depends on who says so in what context. Most medical papers say that a correlation is strong when $|r|\geq 0.8$. Since that only explains $0.64$ of the variance, I would call $0.8$ to be moderate correlation, and in most of my work $R^2\geq 0.95$ with $<5\%$ unexplained variance is called good. In some of my biological work $R^2>0.999$ is required for proper results. On the other hand, for experiments with only short $x$-axis data ranges, and copious noise, one is lucky to even get a significant correlation usually circa $0.5$ as a borderline significant (non-zero) result.

Perhaps the best way to communicate how variable the answer is, is to back calculate what the critical $r$ and $r^2$ values are for a $p<0.05$ significance.

First to calculate the t-value from an r-value let us use

$t=\frac{r}{\sqrt{(1-r^2)/(n-2)}}$, where $n\geq 6$

Then $r=\frac{t}{n-2+t^2}$, where $n\geq 6$ and using the t-significance tables

the critical two-tailed values of $r$ for significance are:

n   r       r^2
6   0.9496  0.9018
7   0.8541  0.7296
8   0.7827  0.6125
9   0.7267  0.5281
10  0.6812  0.4640
11  0.6434  0.4140
12  0.6113  0.3737
13  0.5836  0.3405
14  0.5594  0.3129
15  0.5377  0.2891
16  0.5187  0.2690
17  0.5013  0.2513
18  0.4857  0.2359
19  0.4715  0.2223
20  0.4584  0.2101
21  0.4463  0.1992
22  0.4352  0.1894
23  0.4249  0.1806
24  0.4152  0.1724
25  0.4063  0.1650
26  0.3978  0.1582
27  0.3899  0.1520
28  0.3824  0.1462
29  0.3753  0.1408

30  0.3685  0.1358
40  0.3167  0.1003
50  0.2821  0.0796
60  0.2568  0.0659
70  0.2371  0.0562
80  0.2215  0.0491
90  0.2086  0.0435
100 0.1977  0.0391

Note that the explained fraction ($r^2$) need for a significant $r$-value varies from 90% for $n=6$ to 3.9% for $n=100$. Nor does it stop there, the higher the value of $n$, the less explained fraction is needed for significance.

Finally, asking what a 'good' $R^2$ is, is also a bit ambiguous. Unlike $r^2$, $R^2$ can (surprise, shock and awe) actually take on negative values. So, although $R^2$ is more general than $r^2$, it also has problems that never occur with $r^2$. Moreover, like $r$ (see above), $R$ is $n$ biased, and if we adjust $R$ for degrees of freedom using adjusted $R^2$, negative $R^2$ values become even more frequent.

$\endgroup$
1
$\begingroup$

Consider using the precision and recall scores of scikit-learn: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html. It may give you a more tangible number to consider.

Precision and recall are defined as:

The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative. The recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.

I often too find the score function of the classifiers to be somewhat abstract/ not applicable to my usecase, but the precision and recall gives you a percentage of how many of the predicted items was actually predicted correctly, and how many did the classifier miss.

$\endgroup$
  • $\begingroup$ One of my main issues with sklearn is that it emposes a threshold of 0.5 and only 0.5 on the probabilities predicted from the model when doing classification. In my opinon, this makes much of the build in model tuning features like .score useless. $\endgroup$ – Matthew Drury Mar 7 '17 at 2:14
  • $\begingroup$ @MatthewDrury In that case, do you consider my answer below to be contributory or should I just move it elsewhere to one of the other links listed below the Q above? $\endgroup$ – Carl Mar 8 '17 at 0:09
  • $\begingroup$ No, your answer is good! I should have clarified, it seems reasonable in the regression case, but in the classification it lulls people into using hard classification based on a naive threshold. The answer above is talking about classification, precision, and recall, which I don't like sklearns handling of. $\endgroup$ – Matthew Drury Mar 8 '17 at 0:38
0
$\begingroup$

The Coefficient of Determination (R^2) generalizes the correlation coefficient (r) to multiple predictors, and is often summarized as the proportion of variance explained by the model. It will be quite comfortable for anyone used to analyzing linear regression models, and will be discussed in any text or course you might have takem.

1.0 is a perfect score, good is relative.

Note that the answer which discusses precision and recall does not answer the question posed, which was about Support Vector Regression, not Support Vector Classification. True Positive and False Positive assume binary (True/False) responses.

$\endgroup$
  • 2
    $\begingroup$ If I had had time I would have written something similar to Carl's answer. I can't comment or downvote (not enough reputation points) but felt it was important to point out that the wrong classifier was discussed in wonderkid2's answer. Maybe someone can downvote wonderkid2 and upvote Carl. $\endgroup$ – hwrd Mar 7 '17 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.