1
$\begingroup$

I've implemented a Softmax regression algorithm in java as part of an Android machine learning program. However, no matter how long I let it run for, the accuracy gets to about 70.5% and then plateaus indefinitely, yet the site I got my data from stated that I should be getting close to 90%. I've gone through my code over and over and have been unable to find the source, so I was hoping that you would be able to help. I don't know why it would successfully advance to about 70% and then stop. This is the Softmax formula that I'm using and this is the dataset I'm using

Here is my softmax algorithm:

public List<Double> gradient(List<Double> weights, double[] features, int type){
        int D = Constants.featureSize;
        int K = Constants.numberOfClasses;

        List<Double> grad = new ArrayList<Double>(D*K);
        for(int i = 0; i < D*K; i++){
            grad.add(i, 0.0);
        }
        //Σ(i:k) exp(Θ_i · X) 
        double dot = 0;
        double denom = 0;
        for(int i = 0; i < K; i++){
            //dot product w_i*x
            dot = 0;
            for(int j = 0; j < D; j++){
                dot += features[j] * weights.get(j + (D*i));
            }

            denom += Math.exp(dot);
        }

        //regularization constants
        double[] regular = new double[D * K];
        for(int i = 0; i < D * K; i++){
            regular[i] = 2 * weights.get(i) * Constants.regularizationConstant;
        }

        double prob;
        //prob_i = exp(Θ_i · X)/denom
        for(int i = 0; i < K; i++) {
            //dot product w_i·x
            dot = 0;
            for (int j = 0; j < D; j++) {
                dot += features[j] * weights.get(j + (D * i));
            }
            prob = Math.exp(dot) / denom;

            //∇_0_i = -X(1{i = y} - prob_i)
            int match = 0;
            if(i == type){
                match = 1;
            }
            for (int j = 0; j < D; j++) {
                grad.set(j + (D * i), -1 * features[j] * (match - prob));
            }
        }

        //apply regularization
        for(int i = 0; i < D * K; i++){
            grad.set(i, grad.get(i)+ regular[i]);
        }

        return grad;
    }

Here is where I apply my gradient to the weight (written as a javascript server):

for (i = 0; i < length; i++) { 
                adaG[i] += gradient[i]*gradient[i];
                newWeight[i] = currentWeight[i] - ((c/Math.sqrt(adaG[i]+eps)) * gradient[i]);
            }

weight = newWeight

And then finally, in case you think it might be an error in my accuracy test, here's that code:

var correct = 0;
        var error = 0;
        var labels = require('fs').readFileSync(testLabels).toString().split('\n')
        var features = require('fs').readFileSync(testFeatures).toString().split('\n')
        for(i = 0; i < N; i++){
            var classResults = [];
            line = labels[i];
            var label = parseFloat(line, 10);
            line = features[i];
            var featureStr = line.split(/,| /);
            function valid(str) {
                    return str != "";}
            var featureClean = featureStr.filter(valid);
            var featureArray = [];
            for(var j=0; j<featureClean.length; j++) { 
                featureArray[j] = parseFloat(featureClean[j], 10);}

            for(h = 0; h < K; h++){
                dot = 0;        
                for(j = 0; j < D; j++){
                    dot += featureArray[j]*testWeight[j + (h*D)];}
                classResults[h] = dot;      
            }
            var bestGuess = 0;
            for(h = 0; h < K; h++){
                if(classResults[h]>classResults[bestGuess]){
                    bestGuess = h;}
            }

            if(bestGuess == label){
                correct++;}
            //else{
                //console.log(classResults)
                //console.log('correct: ', label)}      
        }

        var accuracy = correct/N;
        console.log(accuracy)

If anyone could give me some idea of where i'm making an error, please tell me.

$\endgroup$
  • $\begingroup$ A good way to debug would be to test on the train data and achieve close to 100% accuracy. Keep regularization zero for this experiment. If you can't do this, then your network surely has some bug. $\endgroup$ – Parag S. Chandakkar Jun 13 '16 at 0:21
  • $\begingroup$ Even when I get rid of regularization and test on the training set, I'm still getting stuck at 70%, so there's definitely some sort of error going on, but I can't find it. $\endgroup$ – user2785277 Jun 13 '16 at 2:18
  • $\begingroup$ Another trick is to keep simplifying your network till you find the bug. For example, you can just assign equal weights to all the classes (if that's what your testWeight example means, it's not defined in the code). $\endgroup$ – Parag S. Chandakkar Jun 13 '16 at 4:51
3
$\begingroup$

After running more tests with different parameters, I found that the issue was that the initial learning rate was set too low. If I'm perfectly honest I'm pretty frustrated with myself that the solution was that simple, but that's all it was.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.