I've watch the canonical Andrew Ng video on the subject but I'm trying to translate those concepts into java, and I'm not quite sure I did it right.

The thing that's confusing me is, I made a toy example where I had just a few inputs which were identical between the training and test set and still the algorithm misclassified some of the instances of the test set. how is that possible if it's correctly implemented?

For stochastic gradient descent we don't calculate the whole gradient, isn't that right? It's just approximated. I think that would look like this:

  double cost, error, hypothesis;
  double[] gradient;
  int p, iteration;
  
  //Randomly shuffle examples in the training set.
  feature_matrix__train = shuffleArray(feature_matrix__train);

  iteration = 0;
  do 
  {
    iteration++;
    error = 0.0;
    cost = 0.0;
    
    //loop through all instances (complete one epoch)
    for (p = 0; p < number_of_files__train; p++) 
    {
        
      // 1. Calculate the hypothesis h = X * theta
      hypothesis = calculateHypothesis( theta, feature_matrix__train, p, globo_dict_size );

      // 2. Calculate the loss = h - y and maybe the squared cost (loss^2)/2m
      cost = HingeLoss.deriv(hypothesis, outputs__train[p]);

      // STOCHASTIC
      
      //update weights and bias
      for (int i = 0; i < globo_dict_size; i++) 
      {
          theta[i] -= ( LEARNING_RATE * cost * feature_matrix__train[p][i] );
      }
      theta[ globo_dict_size ] -= ( LEARNING_RATE * cost );
      

      //summation of squared error (error value for all instances)
      error += cost;
      
    }
      
  /* Error */
  System.out.println("Iteration " + iteration + " : RMSE = " + error);

  } 
  while( error != 0.0 );

I think my implementation of batch gradient descent is correct, the F1 measure when I run it is generally much better, it looks like this:

  double cost, error, hypothesis;
  double[] gradient;
  int p, iteration;
  
  //Randomly shuffle examples in the training set.
  feature_matrix__train = shuffleArray(feature_matrix__train);

  iteration = 0;
  do 
  {
    iteration++;
    error = 0.0;
    cost = 0.0;
    
    //loop through all instances (complete one epoch)
    for (p = 0; p < number_of_files__train; p++) 
    {
        
      // 1. Calculate the hypothesis h = X * theta
      hypothesis = calculateHypothesis( theta, feature_matrix__train, p, globo_dict_size );

      // 2. Calculate the loss = h - y and maybe the squared cost (loss^2)/2m
      cost = HingeLoss.deriv(hypothesis, outputs__train[p]);
      
      // 3. Calculate the gradient = X' * loss / m
      gradient = calculateGradent( theta, feature_matrix__train, p, globo_dict_size, cost, number_of_files__train);
      
      // BATCH
      
      double[] temp = new double[ globo_dict_size + 1 ];//one for bias
      
      // populate temp to facilitate simultaneous update
      for (int i = 0; i < globo_dict_size; i++) 
      {
          temp[i] = theta[i] - (LEARNING_RATE * gradient[i] );
      }

      // 4. Update the parameters theta = theta - alpha * gradient
      for (int i = 0; i < globo_dict_size; i++) 
      {
          theta[i] = temp[i];
      }

      
      //summation of squared error (error value for all instances)
      error += cost;
      
    }
  
  /* Error */
  System.out.println("Iteration " + iteration + " : RMSE = " + error);

  } 
  while( error != 0.0 );

common methods:

static double calculateHypothesis( double[] theta, double[][] feature_matrix, int file_index, int globo_dict_size )
{
    double hypothesis = 0.0;

     for (int i = 0; i < globo_dict_size; i++) 
     {
         hypothesis += ( theta[i] * feature_matrix[file_index][i] );
     }
     //bias
     hypothesis += theta[ globo_dict_size ];

     return hypothesis;
}

// 3. Calculate the gradient = X' * loss / m
static double[] calculateGradent( double theta[], double[][] feature_matrix, int file_index, int globo_dict_size, double cost, int number_of_files__train)
{
    double m = number_of_files__train;

    double[] gradient = new double[ globo_dict_size + 1 ];//one for bias?
    
    for (int i = 0; i < globo_dict_size; i++) 
    {
        gradient[i] = (1.0/m) * cost * feature_matrix[ file_index ][ i ] ;
    }
    gradient[ globo_dict_size ] = (1.0/m) * cost;
    
    return gradient;
}



/**
 * Computes the first derivative of the HingeLoss loss
 *
 * @param pred the predicted value
 * @param y the target value
 * @return the first derivative of the HingeLoss loss
 */
public static double deriv(double pred, double y)
{
    if (pred * y > 1)
        return 0;
    else
        return -y;
}