I got a data-set with 50 different classes. Around 40000 instances and 48 features(attributes), features are statistical values. I am using weka tool to train and test a model that can perform classification. I have divide my dataset into train and test datasets. 70% of each class name is written into train dataset. 30% for test dataset. For example, if there are 3 instances of class AAA as shown in below sample, then 2 rows (3 x 0.7) of AAA is written to train dataset and remaining 1 row to test data-set. I have train the model using training dataset and the model is re-evaluated using test dataset.
class feature1 feature2 feature 3 featureN AAA 4.6 732 -98 0 AAA AAA BBB BBB CCC CCC CCC CCC CCC
Also I used the whole dataset (without splitting to test and train) to perform cross validation.
When I use 10 fold cross validation I get high accuracy. But with percentage split very low accuracy.
I am not sure if I should use 10 fold cross validation or percentage split for model training and testing? Please advice