I am working on a website where I collect the results of chess games that people have played. Looking at the ratings of the player and the difference between their rating and that of their opponent, I plot a graph with dots representing win (green), draw (blue), and loss (red).
With this information, I also implemented a logistic regression algorithm to categorize the cutoffs for winning and win/drawing. Using the rating and the difference as my two features, I get a classifier, and then draw the boundaries on the chart for where the classifier changes its prediction. Since I've never done this before, I used a pretty basic logistic regression, where I just initialize theta to all zeroes and then run gradient descent on one training set and use the results as is.
Here is my pseudocode for gradient descent. Alpha is the learning rate, J is the cost function, and Theta is the coefficient vector (I think that's what it's called). h is the hypothesis function, and m is the number of training examples.
alpha = 1.0 do 100 times: create a temporary theta array for each feature: take the sum over all the examples of (h(xi) - yi) * xi_j tempTheta_j = theta_j - (alpha/m * sum) Set theta = tempTheta Calculate J = (-1/m) * sum over all examples of ((yi * log(h(xi)) + ((1 - yi) * log(1 - h(xi)))) If J increased, divide alpha by 10. If we're on the 100th run and J decreased by more than 0.001, do 20 more runs. end h(x) = 1/(1 + e^-(Theta^T * xi))
When I test this on the data set representing my own chess profile, I get reasonable results that I can be happy with:
For a while, I was happy. All the examples I tried gave interesting charts. Then I tried a player, Kevin Cao, who had over 250 tournaments to his name, and therefore 1000+ games, for a very large training set. The result was obviously incorrect:
Well, that was no good. So I increased the initial learning rate from 1.0 to 100.0 as my first idea. That got what looks like the right results for Kevin:
Unfortunately, when I then tried it on myself and my smaller data set, I got the strange phenomenon that it just gave a flat line at 0 for one of the predictions:
I checked theta, and it said it was [[2.3707682771730836], [21.22408286825226], [-19081.906528679192]]. The third training variable (really second, since x_0 = 1) is the difference in ratings, so when the difference is just the tiniest bit positive, the formula for logistic regression goes way negative, and the sigmoid function predicts y = 0. When the difference is just the slightly bit positive, similarly, it jumps way up and predicts y = 1.
I reduced the initial learning rate back to 1.0 from 100.0, and decided to instead try reducing it more slowly. So instead of reducing it by a factor of ten when the cost function increases, I reduced it by a factor of two.
Unfortunately, this didn't change the result for me at all. Even if I increased the number of loops of gradient descent from 100 to 1000, it still kept predicting that wrong outcome.
I'm still quite the beginner to logistic regression (I just finished the machine learning class on coursera and this is my first time attempting to implement any of the algorithms I learned there), so I've reached about the extent of my intuition. If somebody would help me figure out what is going wrong here, what I am doing wrong, and how I can fix it I would be extremely grateful.
EDIT: I also tried it on another data set, which had about 300 data points, and got, once again, a flat green line and a normal blue line. The algorithm is basically the same for both, just some different results for y because I'm doing multi-class classification.