I'm quite new to this and wondering if you can help. I'm looking to create an SVM to determine whether neural data from several electrodes can predict behavioral performance on a task. Does the following sound like a reasonable workflow to perform this? I'll be using Matlab, just due to familiarity. I may have as few as 66 trials. I really appreciate any help and advice:
Get data into samples (e.g. trials) x features/predictors
Add column for classification (probably first column)
Select, for example, 80% of dataset to become testing/tuning data
Create model and tune hyperparameters (using nested cross-validation perhaps) using only this 80% of dataset; ensure model is implementing standardization and stratification of classes
Apply tuned model to final 20% test data
Shuffle data (rows) n times (e.g. 1000) to randomise 20% test data and then predict again each time - don't retrain model, just predict using tuned model.
Average predictions and get distribution across shuffled dataset - this is the performance of the model and therefore gives me a description of how well the neural data predict the behavioral response
I originally used the scripted output from classificationLearner in MATLAB with k-fold CV to develop the model based on all the data, and used the validationAccuracy output as my performance measure, but the more I think about it I'm not sure this approach was accurate. Thank you!