How to pick the best mapping matrix after cross validation?

Question

I have an optimization algorithm which finds a mapping matrix from the training data and the i check the performance of that mapping using the test data and a simple classifier like KNN. So that classification accuracy shows the mapping performance.

I evaluate the algorithm performance for each database using 10-fold cross validation with 20 repetitions, which results in 200 different matrices at the end. So how should i pick the best mapping matrix as the outcome of my algorithm for each dataset? for example in order to give it to the costumer!

BTW,The algorithm does need the test data to stop the optimization, so with a different test/train batch a different mapping can be resulted.

1- The average of all 200 matrices wouldn't result in anything useful/correct. 2- Should i just find a matrix based on all the data used as the training data for the algorithm? 3- Is there any specific branch in the literature for combining/using different mapping for the same data and do you recommend that?

Thank you very much :)

Answer 1

I think you should specify what is the performance metric, as in what is the goal of the mapping. The cross validation process has randomness so it makes sense that for 20 repetitions of cross validation you get 20 different results. The point of cross validation is not tell you which result is the best one.

Cross validation is a method to measure the performance of a method, you are perform it multiple times and each iteration with a different training and test set because you want to see how the algorithm performs in different scenarios. You should do this to compare, for example, two different versions of the algorithm, and then the one with the higher cross validation performance should be the algorithm you choose to use.

Once you have chosen on an algorithm, then you should take as much training data as you can get (this statement still depends on the context, but in many cases this is true. also note that the training data should be as 'good' as possible) and then run your algorithm and your 'product' is the output.