Finding best weights for ranking I have a problem concerning Data Science and Machine Learning, and maybe somebody could share a hint on how to accomplish or where to begin with. Thanks in advance. The thing is I have an application that crunches a similarity ranking for every item in the system. The ranking is based on a set of customizable weights that apply to every feature of the item. Currently these weights are defined ad hoc.
In order to change the ranking (because I think certain items should be on top and others on bottom, instead of where they are actually) I need to change those weights. But I want to do that using Machine Learning, with no human interaction. I am being thinking about using perturbation theory and monte carlo stochastic methods to find those perfect ideal weights... Do you think this is the right thing to do? Any alternative ideas? Thanks a lot!
 A: tl;dr
If you have enough example pairs such that you know the characteristics of both samples in the pairs and the rank preferences (A should have higher rank than B), you can pose it as a learning to rank problem.
Details
For a given sample with inputs represented by vector say X, your current weighting procedure is doing dot product X*W and the resulting single number is used to decide the ranking. Now you want to find W such that most of the rankings agree with your intuition. So you could pose this as a learning to rank problem. 
Though, you'll have to study learning to rank a bit on what your feature vectors should represent. Most of the literature that I've studied so far typically has a pair of information on the input side. For example, in learning to rank documents for a given query, the X or independent variables correspond to various similarity calculations between document and query. From the information that you've provided, it appears there's no equivalent query in your case, you are just trying to arrive at a static mapping.
On the part of doing small changes to adapt the weights, you can further use Bayesian approach such that the current weight values are given as priors to learning phase. Every time you have newer data, you re-run the fitting algorithm such that the posterior gets updated.
Hope this helps.
