Problem with calculating $R^2$

Question

I believe I have rather simple question but I would like to make it right.
I have already asked question, however I am not sure whether I did everything correct or there is a mistake in the answer (probably the first one, but still I cannot find it).
To make it short: I would like to check how well my model predicts outcome of a game between two players - and that is why I am trying to calculate R^2. What I have done is based on this answer.
Firstly, I calculate ESS:

ess = 0
for each game in my database:
   ess += (predictedOutcome - realOutcome)^2

The real outcome is either 1 (if win), 0.5 (if draw) or 0 (if lose). To calculate what was the average real outcome, I do:

 averageRealOutcome = (wins + 1/2*draws)/(wins + draws + loses)

The result is always about 0.5 And then for each game I do a loop

tss = 0;
for each win
   tss += (1 - averageRealOutcome)^2
for each draw
   tss += (0.5 - averageRealOutcome)^2
for each lose
   tss += (averageRealOutcome)^2

Then, once more according to the previous answer, I try to calculate R^2:

R^2 = ESS/TSS

Unfortunately the value that I get is above 1, while I believe it should be between 0 (very bad) to 1 (cool). Adjusting R^2 does not do the trick and i can see no difference at all.

1. Does it in general make sense what I try to do?
2. If it does, where is my mistake? Or maybe my interpretation is bad? If not, can you provide and resources where I can read what should I do?

Answer 1

• See http://en.wikipedia.org/wiki/Coefficient_of_determination
• Be careful about notation. Use tot, reg, err.
• From your first code snippet, ess is SS_err. But in the last code snippet, ESS is SS_reg.
• The actual definition of R2 is 1 - SS_err / SS_tot.
• If you use this definition and get R2 < 0, that's not a problem. R2 is only guaranteed to be between 0 and 1 for linear models.
• Use this definition of R2 or a generalization of R2 (also in the Wikipedia article).

Answer 2

Since in your predictive model the design is very like in discrete choice models within the case of binary response variables (logistic regression for example would be relevant here), the problem could be approached in the same manner.

First of all you may consider the generalized $R^2$. Note the last suggestion to scale the $R^2$ by the $R^2_{max}$ that insures the modified coefficient is below $1$.

You may also consult the classical text book in econometrics (I used Greene Econometric Analysis Chapter 21.4.5, but a link from SAS help page seems also relevant) regarding the goodness-of-fit for binary response models:

• McFadden's likelihood ratio index $LRI = 1 - \frac{\log L(\theta)}{\log L(0)}$ the coefficient is pointed to have no clear interpretation though, but this one is a common build-in solution in most of the statistical software
• Another interesting suggestion is to look either on the average probability of correct predictions as in Ben-Akiva and Lerman: $R^2_{BL} = 1/n \sum_i (y_i \hat F_i - (1- y_i)\hat F_i)$, where $\hat F_i$ is the estimated probability (the estimated value you use to compare with the threshold $0.5$).
• In unbalanced samples when the number of wins is much different from the number of losses the $R^2_{BL}$ underestimates the smaller group, so you may introduce a measure that compares average correct and incorrect cases $\lambda = (average \hat F_i | y_i = 1) - (average \hat F_i | y_i = 0)$ Cramer's $\lambda$ heavily penalizes incorrect answers.
• Note that for the unbalanced datasets it is recommended to be cautious about the naive threshold value of $0.5$ since in that case a smaller group risks to be unpredicted too often.
• The other suggestions for the $R^2$ augmentation are in the abovementioned SAS help page (to be frank, I have never used one thus has no clear opinion regarding them, but the Cragh-Uhler 1 is exactly the generalized coefficient of determination from the wiki page).