Logistic Regression failing in some cases [closed]

I am working on a website where I collect the results of chess games that people have played. Looking at the ratings of the player and the difference between their rating and that of their opponent, I plot a graph with dots representing win (green), draw (blue), and loss (red).

With this information, I also implemented a logistic regression algorithm to categorize the cutoffs for winning and win/drawing. Using the rating and the difference as my two features, I get a classifier, and then draw the boundaries on the chart for where the classifier changes its prediction. Since I've never done this before, I used a pretty basic logistic regression, where I just initialize theta to all zeroes and then run gradient descent on one training set and use the results as is.

Here is my pseudocode for gradient descent. Alpha is the learning rate, J is the cost function, and Theta is the coefficient vector (I think that's what it's called). h is the hypothesis function, and m is the number of training examples.

alpha = 1.0
do 100 times:
create a temporary theta array
for each feature:
take the sum over all the examples of (h(xi) - yi) * xi_j
tempTheta_j = theta_j - (alpha/m * sum)
Set theta = tempTheta
Calculate J = (-1/m) * sum over all examples of ((yi * log(h(xi)) + ((1 - yi) * log(1 - h(xi))))
If J increased, divide alpha by 10.
If we're on the 100th run and J decreased by more than 0.001, do 20 more runs.
end

h(x) = 1/(1 + e^-(Theta^T * xi))

When I test this on the data set representing my own chess profile, I get reasonable results that I can be happy with: For a while, I was happy. All the examples I tried gave interesting charts. Then I tried a player, Kevin Cao, who had over 250 tournaments to his name, and therefore 1000+ games, for a very large training set. The result was obviously incorrect: Well, that was no good. So I increased the initial learning rate from 1.0 to 100.0 as my first idea. That got what looks like the right results for Kevin: Unfortunately, when I then tried it on myself and my smaller data set, I got the strange phenomenon that it just gave a flat line at 0 for one of the predictions: I checked theta, and it said it was [[2.3707682771730836], [21.22408286825226], [-19081.906528679192]]. The third training variable (really second, since x_0 = 1) is the difference in ratings, so when the difference is just the tiniest bit positive, the formula for logistic regression goes way negative, and the sigmoid function predicts y = 0. When the difference is just the slightly bit positive, similarly, it jumps way up and predicts y = 1.

I reduced the initial learning rate back to 1.0 from 100.0, and decided to instead try reducing it more slowly. So instead of reducing it by a factor of ten when the cost function increases, I reduced it by a factor of two.

Unfortunately, this didn't change the result for me at all. Even if I increased the number of loops of gradient descent from 100 to 1000, it still kept predicting that wrong outcome.

I'm still quite the beginner to logistic regression (I just finished the machine learning class on coursera and this is my first time attempting to implement any of the algorithms I learned there), so I've reached about the extent of my intuition. If somebody would help me figure out what is going wrong here, what I am doing wrong, and how I can fix it I would be extremely grateful.

EDIT: I also tried it on another data set, which had about 300 data points, and got, once again, a flat green line and a normal blue line. The algorithm is basically the same for both, just some different results for y because I'm doing multi-class classification.

closed as off-topic by kjetil b halvorsen, mdewey, Peter Flom♦Aug 13 '18 at 9:05

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – kjetil b halvorsen, mdewey, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

• This question basically asks us to debug your code, which evidently is not working as expected. That's a lot to ask--this is not a code review site. You can get more effective help by telling us what efforts have you have made to test and verify the code, by describing its algorithm, and by isolating the problematic section as much as possible. – whuber Dec 6 '12 at 18:50
• Fair enough--but to get started, we have to read through your code in detail just to determine what algorithm you are using and what precisely you might mean by "logistic regression." (That automatically eliminates at least 90% of your audience, I would guess, because you are limiting your communication to those who know Ruby--a pretty small subset--or those intrepid readers willing to guess what it's doing.) It would be easier on would-be answerers if you were to describe the procedure in English and mathematical langauge: somebody might then immediately see what's going on. – whuber Dec 6 '12 at 19:02
• Why are you writing your own algorithm? Why not just run an ordinal logistic regression where "result" (W/D/L) is the DV and rating and difference are the IVs? – Peter Flom Dec 6 '12 at 22:11
• I'm writing my own to try to learn it better and develop some intuition by trying to make mine good. – Andrew Latham Dec 7 '12 at 0:02
• That's fine, Andrew, but the right way to go about it is to begin with working software (or at least a worked-out test dataset in a textbook or paper) and apply your code to simple datasets in order to reproduce known results. I know (from ample experience) that it's unwise to plunge ahead to apply your own code to actual data until such testing has been thoroughly carried out. – whuber Dec 7 '12 at 14:38