I have an optimization algorithm which finds a mapping matrix from the training data and the i check the performance of that mapping using the test data and a simple classifier like KNN. So that classification accuracy shows the mapping performance.
I evaluate the algorithm performance for each database using 10-fold cross validation with 20 repetitions, which results in 200 different matrices at the end. So how should i pick the best mapping matrix as the outcome of my algorithm for each dataset? for example in order to give it to the costumer!
BTW,The algorithm does need the test data to stop the optimization, so with a different test/train batch a different mapping can be resulted.
1- The average of all 200 matrices wouldn't result in anything useful/correct. 2- Should i just find a matrix based on all the data used as the training data for the algorithm? 3- Is there any specific branch in the literature for combining/using different mapping for the same data and do you recommend that?
Thank you very much :)