# MNIST Softmax regression hitting a wall at 70% accuracy

I've implemented a Softmax regression algorithm in java as part of an Android machine learning program. However, no matter how long I let it run for, the accuracy gets to about 70.5% and then plateaus indefinitely, yet the site I got my data from stated that I should be getting close to 90%. I've gone through my code over and over and have been unable to find the source, so I was hoping that you would be able to help. I don't know why it would successfully advance to about 70% and then stop. This is the Softmax formula that I'm using and this is the dataset I'm using

Here is my softmax algorithm:

public List<Double> gradient(List<Double> weights, double[] features, int type){
int D = Constants.featureSize;
int K = Constants.numberOfClasses;

for(int i = 0; i < D*K; i++){
}
//Σ(i:k) exp(Θ_i · X)
double dot = 0;
double denom = 0;
for(int i = 0; i < K; i++){
//dot product w_i*x
dot = 0;
for(int j = 0; j < D; j++){
dot += features[j] * weights.get(j + (D*i));
}

denom += Math.exp(dot);
}

//regularization constants
double[] regular = new double[D * K];
for(int i = 0; i < D * K; i++){
regular[i] = 2 * weights.get(i) * Constants.regularizationConstant;
}

double prob;
//prob_i = exp(Θ_i · X)/denom
for(int i = 0; i < K; i++) {
//dot product w_i·x
dot = 0;
for (int j = 0; j < D; j++) {
dot += features[j] * weights.get(j + (D * i));
}
prob = Math.exp(dot) / denom;

//∇_0_i = -X(1{i = y} - prob_i)
int match = 0;
if(i == type){
match = 1;
}
for (int j = 0; j < D; j++) {
grad.set(j + (D * i), -1 * features[j] * (match - prob));
}
}

//apply regularization
for(int i = 0; i < D * K; i++){
}

}


Here is where I apply my gradient to the weight (written as a javascript server):

for (i = 0; i < length; i++) {
}

weight = newWeight


And then finally, in case you think it might be an error in my accuracy test, here's that code:

var correct = 0;
var error = 0;
for(i = 0; i < N; i++){
var classResults = [];
line = labels[i];
var label = parseFloat(line, 10);
line = features[i];
var featureStr = line.split(/,| /);
function valid(str) {
return str != "";}
var featureClean = featureStr.filter(valid);
var featureArray = [];
for(var j=0; j<featureClean.length; j++) {
featureArray[j] = parseFloat(featureClean[j], 10);}

for(h = 0; h < K; h++){
dot = 0;
for(j = 0; j < D; j++){
dot += featureArray[j]*testWeight[j + (h*D)];}
classResults[h] = dot;
}
var bestGuess = 0;
for(h = 0; h < K; h++){
if(classResults[h]>classResults[bestGuess]){
bestGuess = h;}
}

if(bestGuess == label){
correct++;}
//else{
//console.log(classResults)
//console.log('correct: ', label)}
}

var accuracy = correct/N;
console.log(accuracy)


If anyone could give me some idea of where i'm making an error, please tell me.

• A good way to debug would be to test on the train data and achieve close to 100% accuracy. Keep regularization zero for this experiment. If you can't do this, then your network surely has some bug. – Parag S. Chandakkar Jun 13 '16 at 0:21
• Even when I get rid of regularization and test on the training set, I'm still getting stuck at 70%, so there's definitely some sort of error going on, but I can't find it. – user2785277 Jun 13 '16 at 2:18
• Another trick is to keep simplifying your network till you find the bug. For example, you can just assign equal weights to all the classes (if that's what your testWeight example means, it's not defined in the code). – Parag S. Chandakkar Jun 13 '16 at 4:51