Does number of positive and negative examples (balance) influence classification performance? I am trying to classify a data set X of 2000 examples (rows) and 20 features (Columns) .
The class labels are 0 or 1. Out of 2000 examples, 98 are labelled as 1 and the rest are 0.
I am using MATLAB 2018 version.
I want to apply cross-validation using HoldOut method and using 60/40 of the data X.  
Question:
 I am getting all predicted classes as 1  and accuracy of 98.1%. Eventhough the accuracy is so high, the predicted class labels are mostly incorrect. 
Where am I going wrong? Is it because out of 2000 examples, only 98 are positive examples (labelled 1) and the remaining 0? 
Is SVM not the proper choice for this kind of data?
 A: If yours is an imbalanced dataset where your class proportion ( no. of minority class samples to no. of majority class samples)  is very low as compared to how it actually should be . 
For example , in  case of the famous Iris Dataset where the classes should be balanced(No. of positive  class samples ≈ No. of negative class samples)  , but , say you have a version of the Iris dataset where there are 120 positive samples and just 10 negative samples , you can try out synthetic undersampling or synthetic oversampling techniques.
In undersampling , you can take a subset of the majority class whose size is comparable to that of minority class.
While, in oversampling, you can replicate the minority class to make its size comparable to that of majority class.
The undersampling and oversampling techniques will not be viable when dealing  with datasets such as Fraud Detection Dataset, where the no. of minority samples are supposed to be very less as compared to that of no. of majority samples , i.e. , the class proportion matches the base rate.
A: The most important question here is: why does the data have this imbalance. 


*

*If the imbalance reflects the base rate* of your classes, you should go with the imbalanced data.

*Resampling to rebalance the data should IMHO be avoided unless you positively know the proper base rates and resample to get there. But applications with equal base rates are very rare in my experience.
So if you know the base rates should be equal but your data doesn't reflect this, go ahead and rebalance. 

*Training and validating on equal base rates will often yield a model with seemingly nice performance - which however often breaks down if evaluated for the real (imbalanced) base rates. 

*As you already noticed, plain acuracy may not be the best figure of merit for unbalanced classes, and

*the naive model to compare your performance to is not 50:50 random guessing but "always predict 0" with 98 % accuracy.

*have a look at your misclassification costs: is missing a positive worse than a false positive? and

*adjust your loss function accordingly.

*Consider only figures of merit that properly take into account class base rates (or are independent of class base rate).  



*

*Classifiers that take class imbalance "naturally" are obviously better suited. However, this is often a question of implementation than of the general capability of the classifier in question. 

*If you cannot "tell" your SVM implementation about base rates and misclassification costs, but it can return a continuous score (e.g. predicted class membership probability) you can request that score and then choose your working point (cutoff score) yourself. E.g. you can choose the cutoff so that the number of false positives = number of false negatives (times factor for respective costs).



Seeing that you have only 98 positive examples, consider resampling validation (stratified cross valiation or out-of-bootstrap) rather than hold out. 

* base rate or prior probability: the fraction of cases truly belonging to the class in question in the relevant population (which is not necessarily the same in the sample at hand).
"Balanced situation" typically refers to all classes having equal prior probability (or base rate) of $\frac{1}{nclasses}$, so 50:50 in case of 2 classes.
