0
$\begingroup$

I'm facing a binary classification problem using svm light. However using 5-fold-validation I noticed that later I train SVM with training set (Half positive and half negative samples about) the predictions on the test set(Half positive and half negative samples about) are almost all positives or all negatives causing an accuracy of 50% about.

What could be the reason for this problem? Thanks in advance!

Update: Added the code of k-fold-validation

double gi::approximateOracle::k_fold_validation(const vector< pair<vector<double> , labelType> > &mapped_samples ,const int * indices ,const int k,const string &C ,const double &gamma)
{
double error;
double accuracy=0.0;
double mean_accuracy;

int num_samples = mapped_samples.size(); //num of samples
int remainder = num_samples % k; //reminder of division num_samples/k
int num_samples_less_remainder = num_samples - remainder;
int base_dimension_one_fold  = num_samples_less_remainder / k; //It's sure that rest of this division is zero. This is the base dimension of test set too
int base_dimension_train_set = base_dimension_one_fold * (k-1); //dimension of one fold multiplied for k-1 (because one fold go in test set)

/*In the k-fold-validation in turn a fold have to be inserted in test set and the others in testing set. Starting from
 * the last (those with index of the fold higher) in the test set, the first k-remainder times the training set dimesion
 * have to be base_dimension_train_set+remainder and the others remainder times the dimension of training set have to be
 * base_dimension_train_set+remainder-1 . The dimension of trainig set, in anlogous way, is the first k-remainder times
 * is base_dimension_one_fold and the other remainder times is base_dimension_one_fold+1
 */
vector< pair<vector<double> , labelType> > train_samples(base_dimension_train_set+remainder);
vector< pair<vector<double> , labelType> > test_samples;
test_samples.reserve(base_dimension_one_fold+1);
test_samples.resize(base_dimension_one_fold);//This operation and that above for avoid reallocation.

for(int num_round = 1 ; num_round<k+1 ; num_round++) //CAUTION the kfold function of gi_utilities.cpp return indices from 0 to k-1
{
    if(num_round == (k-remainder+1))
    {
        train_samples.resize(base_dimension_train_set+remainder-1); //this doesn't cause reallocation because the size is minor than previous
        test_samples.resize(base_dimension_one_fold+1); //this doesn't cause reallocation for the reserve above
    }

    for(int index_samples=0,index_train=0,index_test=0 ; index_samples<mapped_samples.size() ;index_samples++)
    {
        if(indices[index_samples] == (k-num_round)) //in the cycle indices(num_round) go from 1 to k. I must have indices from k-1 to 0 
        {   
           test_samples[index_test] = mapped_samples[index_samples];
           index_test++;
        }
        else
        {
            train_samples[index_train] = mapped_samples[index_samples];
            index_train++;
        }
    }

    write_samples_file_svm_light_format(train_samples,TRAIN_FILE);
    write_samples_file_svm_light_format(test_samples,TEST_FILE);
    error = build_classifier(C , gamma , false); //false i.e. doesn't calculate LOO. return -1 in this case 
    accuracy += testing();
}

mean_accuracy = accuracy / k ;
assert(("L'accuracy media trovata con k-fold-validation non è compresa tra 0 e 100",(mean_accuracy>=0 && mean_accuracy<=100.0)));
return mean_accuracy;
}

and here the functions shuffle and kfold used for create the indices

    /*Implemented following Knuth shuffle. More details at https://en.wikipedia.org/wiki/Fisher%E2%80%93Yates_shuffle */
void shuffleArray(int* array,int size) 
{
  int k;
  int temp; 
  int n = size;
  while (n > 1) 
  {
   // 0 <= k < n.
   k = rand()%n;        

   // n is now the last pertinent index;
   n--;                 

   // swap array[n] with array[k]
   temp = array[n]; 
   array[n] = array[k];
   array[k] = temp;
  }
 }

   //Don't forget to delete the memory returned from this method
   int* kfold( int size, int k )
   {
  int* indices = new int[ size ];

  for (int i = 0; i < size; i++ )
    indices[ i ] = i%k;

  shuffleArray( indices, size );

  return indices;
    }

The indices are created in this way:

    srand(time(0));
    indices = kfold(mapped_samples.size() , 5 ); //this method return dynamic memory

The function build_classifier() do the training (there is there a call to a function of svm_light). testing() use the test set (writed in the file) and the model (writed in another file) for do predictions and return the accuracy and in another file return for each sample of the test set a number that point the prediction is positive or negative (>0 or not)

$\endgroup$
12
  • $\begingroup$ which kernel are you using? what happens if you switch your test and training sets? $\endgroup$ – redress Jun 2 '17 at 19:45
  • $\begingroup$ Does it happen for all 5 validations? If you split the data at random, is the problem still there? $\endgroup$ – Karel Macek Jun 2 '17 at 19:47
  • 1
    $\begingroup$ Have you tried a simpler model first, such as logistic regression? $\endgroup$ – Kodiologist Jun 2 '17 at 19:57
  • $\begingroup$ @redress For now gaussian kernel with grid-search, thus with different parameters. But for a lot of parameters it happen this situation. I have to see what happen in the situation that you told $\endgroup$ – Umbert Jun 2 '17 at 20:37
  • $\begingroup$ @KarelMacek I have to see if for all 5 validations. The data are already splitted at random $\endgroup$ – Umbert Jun 2 '17 at 20:38
0
$\begingroup$

I assume that your cross validation is implemented correctly the first glance at your code does not exhibit anything weird.

I also assume that you use svmlight without modifications and it works correctly.

The only thing that can cause your problems can be the improper selection of parameters. Please check this: https://en.wikipedia.org/wiki/Support_vector_machine#Parameter_selection as well as the documentation of SVM light. Try multiple combinations.

I have not seen your data, but my intuition is that:

  • They are not separable and in combination with that...
  • ...the parameters cause something like majority voting. If the training set has majority of positives, it results into all positives in the test set.

My advice:

$\endgroup$
4
  • $\begingroup$ Yes, I already implemented grid-search using the values of your wikipedia paper, And for most parameters (but not all) happens the behavior described. Instead, data may not be separable. I will try for other data, representing a simpler problem. And to increase the dimension of the training set. The data are balanced. Thanks for advice, I will attempt. I will search too for same kernel specific (in svm light is possible have a personalized kernel) for my problem and for different encode for my data $\endgroup$ – Umbert Jun 5 '17 at 11:14
  • $\begingroup$ If you are happy with the advice, feel free to upvote or accept it. $\endgroup$ – Karel Macek Jun 5 '17 at 12:51
  • $\begingroup$ I cant't upvote because I have less than 15 reputation. I don't accept it because for now the problem is still not resolved $\endgroup$ – Umbert Jun 5 '17 at 19:39
  • $\begingroup$ As the training in some cases stopped I followed the FAQ of SVM_LIGHT at cs.cornell.edu/People/tj/svm_light/svm_light_faq.html where it is spoken of slow convergence and you tells of limit C (cost factor). But in this way svm C the penalty decreases and it may be cause the error.In fact I augmented C and the problem descived above there was more (predictions all positives or all negatives) Although accuracy has not improved much (But this may depend from encode,kernel,scaling...)Although not limiting C as it is said in the FAQ, the convergence process can become very very slow $\endgroup$ – Umbert Jun 6 '17 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.