I have a data set which consists of 10 numerical variables, as well as a rank ordering of the observations. I am "scoring" each observation with a weighted average of the numerical variables, where the weights are chosen so that the resulting Kendall correlation coefficient of the score with the rank ordering is maximised. (All numerical variables have values between 0 and 100.) I'm using a genetic algorithm to find the optimal weights (which must sum up to 1).
In order to assess the out-of-sample properties of the model, I cross-validate as follows. I obtain optimised weights using all but one observation as training set, and then calculate the score for the last observation. Using scores obtained in this way for all observations I calculate the Kendall's Tau with the rank ordering. A better Kendall's Tau means a better model.
I am worried about comparing the scores obtained with different weights for each observation. Do I need to adjust the out-of-sample scores based on properties of each training data set?