Optimal weights to maximize similarity of two ordinal rankings

Question

I want to find the optimal weights so that the pairwise Euclidean distances in my training dataset are closest possible to the differences in ordinal rankings. In other words, I want to train the weights to increase the rate of correct classifications based on nearest neighbour. There are a numbers of issues here: (1) which distance metric is best to evaluate the similarity of two ordinal rankings, (2) how best to transform the pairwise Euclidean distances into a comparable ordinal ranking, and (3) which optimization model to use.

The objective of this exercise is to provide a simple and intuitive method to improve correct classifications in a clustering algorithm. I do not want to add further assumptions to the model at this stage, for instance by defining the number of clusters.

For (1) I have used the Normalized Discounted Cumulative Gain as the simpler Kendall distance does not penalize for large deviations between the rankings. The issue with the current code is that the chosen optimization model fails to find a global optimum (3), in part due to issues on how the scaling is applied in (2) that provides a biased output in (1).

Let's assume a training dataset 'data' containing in the first column the ids of the entities to classify, in the second their ordinal ranking, and in the p+2 columns their features.

library(rrecsys)

WF <- function(weights, data) {   ## function to optimise
  weightedData <- as.matrix(data[,-c(1,2)]) %*% diag(weights) # weigh each dimension
  distMatrix <- as.matrix(dist(weightedData, method = "euclidean")) # Euclidean distance matrix
  distMatrix <- stack(as.data.frame(distMatrix)) # long format
  rankMatrix <- matrix(data[,2],nrow=dim(data)[[1]],ncol=dim(data)[[1]],byrow=TRUE) # turn ordinal rankings into pairwise matrix
  rankDifferentiationMatrix <- abs(rankMatrix-t(rankMatrix)) # substract pairwise and create new ranking on relative distances
  rankDifferentiation <- stack(as.data.frame(rankDifferentiationMatrix)) # long format
  output <- as.data.frame(cbind(rankDifferentiation[,1], distMatrix[,1])) # merge
  output<- output[which(output[,1] != 0 | output[,2] != 0),] # remove diagonal
  output <- unique(output) # remove upper triangle
  output[,1] <- round(((1-output[,1])^2)^(0.5)/max(output[,1]),2) # scale ranking to 0-1
  output[,2] <- round(((1-output[,2])^2)^(0.5)/max(output[,2]),2) # scale pairwise distances to 0-1
  DCG <- eval_nDCG(output[,1], output[,2]) # Normalized Discounted Cumulative Gain
  return(1-DCG)
}

OptWeights <- optim(par=rep(1,p),fn=WF, data=data) # Find optimal weights