Cross validation or percentage split

I got a data-set with 50 different classes. Around 40000 instances and 48 features(attributes), features are statistical values. I am using weka tool to train and test a model that can perform classification. I have divide my dataset into train and test datasets. 70% of each class name is written into train dataset. 30% for test dataset. For example, if there are 3 instances of class AAA as shown in below sample, then 2 rows (3 x 0.7) of AAA is written to train dataset and remaining 1 row to test data-set. I have train the model using training dataset and the model is re-evaluated using test dataset.

class  feature1  feature2 feature 3 featureN
AAA    4.6       732      -98       0
AAA
AAA
BBB
BBB
CCC
CCC
CCC
CCC
CCC


Also I used the whole dataset (without splitting to test and train) to perform cross validation.

When I use 10 fold cross validation I get high accuracy. But with percentage split very low accuracy.

I am not sure if I should use 10 fold cross validation or percentage split for model training and testing? Please advice

• Are you asking about stratified sampling? Or maybe you have high accuracy in the bigger classes but low in the smaller ones?+ – user2974951 Aug 6 at 6:09