I'm currently following this lesson: http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/ , and after I coded their equation I got:

Training accuracy: 71.2%
Test accuracy: 72.0%

Here is the relevant piece of code:

% function [f,g] = softmax_regression(theta, X,y)
function [f, g] = softmax_regression_vec(theta, X, y)
  % Arguments:
  %   theta - A vector containing the parameter values to optimize.
  %       In minFunc, theta is reshaped to a long vector.  So we need to
  %       resize it to an n-by-(num_classes-1) matrix.
  %       Recall that we assume theta(:,num_classes) = 0.
  %   X - The examples stored in a matrix.  
  %       X(i,j) is the i'th coordinate of the j'th example.
  %   y - The label for each example.  y(j) is the j'th example's label.

  % theta is a vector;  need to reshape to n x num_classes.
  theta=reshape(theta, n, []);

  % initialize objective value and gradient.
  f = 0;
  g = zeros(size(theta));

  % TODO:  Compute the softmax objective function and gradient using vectorized code.
  %        Store the objective function value in 'f', and the gradient in 'g'.
  %        Before returning g, make sure you form it back into a vector with g=g(:);
  sequence = 1 : num_classes ;
  sequence = sequence';
  for i = 1 : m
      % Objective function
      expo = exp(theta' * X(:,i));
      totalSum = sum(expo) + 1;
      P = [expo / totalSum ; 1 / totalSum];
      f = f - log(P(y(i))) ;      

      % Gradient
      diff = (y(i) == sequence) - P;
      tmp = bsxfun(@times, X(:,i), diff' );      
      tmp = bsxfun(@plus, tmp, tmp(:,num_classes));       
      g = g - tmp(:,1 : num_classes - 1) ;

  g=g(:); % make gradient a vector for minFunc

Is it unusual for softmax to achieve 70% ? Please help me thank you very much

  • $\begingroup$ what's the data? MNIST? what make you think it can't achieve such accuracy? what make your think 70% is any good? $\endgroup$
    – horaceT
    Commented Jul 1, 2016 at 16:42
  • $\begingroup$ @horaceT in here: ufldl.stanford.edu/wiki/index.php/… they say they achieved 90% accuracy, while mine got only 70% :( $\endgroup$ Commented Jul 1, 2016 at 16:43
  • $\begingroup$ I couldn't post a comment. That's why I ask my question here: Why are you doing this line? (Adding the last column of Jacobian matrix tmp to all the columns?) tmp = bsxfun(@plus, tmp, tmp(:,num_classes)); $\endgroup$
    – Nickie
    Commented Apr 14, 2019 at 2:41

1 Answer 1


Here are some things you should check that might be holding you back:

  • Use a suitable loss function (i.e., the cross-entropy loss).

  • Use weight decay or regularization (tune the associated hyper-parameter using cross-validation).

  • Since you manually wrote code to compute the code, don't forget to implement a gradient check, to test for bugs in your code.

  • You might want to use an off-the-shelf optimizer rather than implementing gradient descent yourself. This will typically include improvements like momentum, learning rate, L-BGFS, or other techniques.

  • You might want to initialize the weights not to all zeros, but rather to small random numbers (e.g., chosen from a Gaussian), to break symmetry.

  • You might need to tune hyperparameters for your optimization routine (learning rate, momentum, etc.) via cross-validation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.