Hot answers tagged

34

I am actually looking for the same answer, however I should be able to at least partially answer your question. All of the metrics that you have mentioned have different traits and, unfortunately, the one you should pick depends on what you actually would like to measure. Here are some things that it would be worth to have in mind: Spearman's rho metric ...


15

3 years after, I answer to my own question. For me, the main difference is in what is the output of the models in the different problems. In ordinal regression, the task is to predict a label for a given sample, hence the output of a prediction is a label (as is the case for example in multiclass classification). On the other hand, in the problem of ...


13

Yes- there are many versions of multi-participant rating systems, many of which are modified versions of the base Elo system (one of Elo's great strengths is that it can be readily modified). One very interesting multi-participant ranking system is Microsoft's TrueSkill Ranking system based on bayesian inference of player skill. The rating is used in ...


12

Awesome question. Question 1: I approach this problem using standard deviations ($n\sigma$) to create a standardized scale where $n$ is the number of standard deviations from the mean ($\mu$) and $\sigma$ is the standard deviation. I will use an example of a call center agent making calls. Here is a possible way to define the scale using $n$: $m_+ = n$...


9

Hopefully this isn't too late, but your data is still valuable using the Bradley-Terry model. This paper is a nice reference MM Algorithms for generalized Bradley-Terry models. Equation 4 is the one you want. Brief explanation as I guess you are not a mathematition, and I'm being nice as I'd really like a copy of your data to test my implementation of this ...


9

In many cases where you apply ranking algorithms (e.g. Google search, Amazon product recommendation) you have hundreds and thousands of results. The user only wants to watch at the top ~20 or so. So the rest is completely irrelevant. To phrase it clearly: Only the top $k$ elements are relevant If this is true for your application, then this has direct ...


7

As with many such situations, one must take care to avoid confusing sample and population quantities. (We might test for symmetry about a population mean using a statistic based on sample medians for example.) (We should also keep in mind that failure to reject a null of symmetry is not the same as showing symmetry.) Let's begin by simplifying things by ...


7

I am not sure why you were looking at correlations and similar measures. There doesn't seem to be anything to correlate. Instead, there are a number of options, none really better than the other, but depending on what you want: Take the average rank and then rank the averages (but this treats the data as interval) Take the median rank and then rank the ...


7

I would try to put the data into the same chart, not split into different panels, to allow more comparisons. Of course, there is a lot of experimentation with graphics before you decide which show the data best. Here are a handful that I would try. Assuming 1 is most important I might reverse the category axis, take off the numbers, and show Most and Least ...


6

This type of problem is typically referred to in econometrics and marketing research as a "choice modeling" problem. Texts dealing with such problems include: Louviere, J., D. A. Hensher, et al. (2000). Stated Choice Methods: Analysis and Application. Cambridge, Cambridge University Press. Train, K. E. (2009). Discrete Choice Methods with Simulation. ...


6

Summary I share my thoughts in Details section. I think they are useful in identifying what we really want to achieve. I think that the main problem here is that you haven't defined what a rank similarity means. Therefore, no one knows which method of measuring the difference between the ranks is better. Effectively, this leaves us to ambiguously choose a ...


6

Ordinal regression is ideal for this problem in my opinion. There is no problem other than computational burden caused by having as many unique $Y$ as there are observations. The R rms package's orm function solves the computational burden problem using a special sparse matrix representation. For an example see Which model should I use to fit my data ? ...


6

Unlike Spearman's rho, Kendall's tau doesn't actually require assigning numerical rankings to entries. Instead, it functions off of concordant and discordant pairs. For example, say you have the following rankings A B America UK UK France Germany America France Spain Italy Germany An example of a ...


6

This is a great question that people are doing research on. Therefore, part of my answer will be throwing papers at you. Offline vs. Online. (The get more labels approach) I assume that you want to correct these labels so you can do an offline evaluation. One of the strategies for dealing with this issue is an "interleaved" evaluation where you present a ...


5

Let me expand on alternative solution proposed by @curious_cat. $P_{ij}$ is the matrix of pitches $L_{ij}$ is the matrix of sells $S_{ij} = L_{ij}/P_{ij}$ is the matrix of success rates (elementwise division where it exists and 0 elsewhere) As @curious_cat suggested, you want to approximate $S_{ij}$ by the outer product of two positive vectors $$S_{...


5

This sounds like the 'Willcoxon signed-rank test' (wikipedia link). Assuming that the values of your ranks are from the same set (ie [1, 25]) then this is a paired-difference test (with the null-hypothesis being these pairs were picked randomly). NB this is a dis-similarity score! There are both R and Python implementations linked to in that wiki page.


5

[Forewarning: this answer appeared before the OP decided to reformulate the question, so it may have lost relevance. Originally the question was about How to rank items according to their pairwise correlations] Because matrix of pairwise correlations isn't a unidimensional array it is not quite clear what "ranking" may look like. Especially as long as you ...


5

Are you perhaps thinking of MaxDiff, a.k.a., best−worst scaling? MaxDiff sounds like a wonderful method, but do note that it isn’t as amazing as some make it out to be, and it has inherent problems. These blog posts have some information on this: Warning: Sawtooth’s MaxDiff Is Nothing More Than a Technique for Rank Ordering Features! If SPSS can factor ...


5

this information is not enough. Let's look more precisely what I mean by the lack of information. An event $CA$ means that $C$ wins against $A$ and event $\overline{CA}$ for when $C$ looses against $A$. Then we have: \begin{equation} p(CB) = p(CB|CA)p(CA) + p(CB|\overline{CA})p(\overline{CA}) \end{equation} and following that we have, \begin{equation} p(...


5

Xeon is right in what TF-IDF and cosine similarity are two different things. TF-IDF will give you a representation for a given term in a document. Cosine similarity will give you a score for two different documents that share the same representation. However, "one of the simplest ranking functions is computed by summing the tf–idf for each query term". This ...


5

Yes. However, not very seldom rankings are treated as interval data and are analyzed by parametric procedures (such as ANOVA). For example, this is customarily done in classic conjoint analysis. Conceptually, psychometrically, rankings are ordinal while ratings are potentially interval (albeit often cautiously treated as ordinal as well). Still, ...


5

I think the answer you are looking for might be the Boruta algorithm. This is a wrapper method that directly measures the importance of features in an "all relevance" sense and is implemented in an R package, which produces nice plots such as where the importance of any feature is on the y-axis and is compared with a null plotted in blue here. This blog ...


4

Since asking this question, I've found I've had lots of success with the PlayerRatings package for R. It makes creating ELO/Glicko and the authors own method of performance ratings very easy.


4

Why not for each merchant compute a success rate for every product he sells $S_{ij}$. ($i$ indexes products and $j$ indexes merchants) Average this and compute a merchant average baseline success rate($S_j$). Now compute differences ($\delta S_{ij}=S_{ij} - S_j$). Each of this $\delta S_{ij}$ indicates how much better or worse every product does with ...


4

Your problem can be modeled by a Rasch Model. Here is a document that explains the model with the following example Rasch model is a statistical model of a test that attempts to describe the probability that a student answers a question correctly. It assigns to every student a real number, a, called the "ability", and to every questions a real ...


4

Warning: it's a great question and I don't know the answer, so this is really more of a "what I would do if I had to": In this problem there are lots of degrees of freedom and lots of comparisons one can do, but with limited data it's really a matter of aggregating data efficiently. If you don't know what test to run, you can always "invent" one using ...


4

As others have pointed out, there are a lot of options you might pursue. The method I recommend is based on average ranks, i.e., the first proposal of Peter. In this case, the statistical importance of the final ranking can be examined by a two-step statistical test. This is a non-parametric procedure consisting of the Friedman test with a corresponding ...


4

If I understand correctly, what you would like to have is a measure to compare the underlying true ranking $\pi$ and the predicted ranking (i.e., the tournament ranking, or the simulated ranking) $\sigma$, where $\sigma$ is a function of some input parameters. In the statistics literature, there are a number of distance functions for rankings. I will list ...


4

Pr(A > B | A > C) = Pr(A > max(B,C)) / Pr(A > C). Here is Mathematica code for it. The counts are obtained using the same technique used to answer the previous question. na = Length@a; (Tr@Ordering[Ordering@Join[a, Max/@Tuples@{b,c}], na] - na(na+1)/2) / (Length@b * (Tr@Ordering[Ordering@Join[a, c ], na] - na(na+1)/2)) EDIT - Here's some test data: a = ...


4

Usually, scheduling based on seeding seems to be set up to improve the chance to retain the better players. Here's one obvious possibility: Going by seed, if we want to maximize the chances of keeping the top few seeds, we want the highest seed they face to be low and within that, the average seed they face to be fairly low. One approach: Consider that ...


Only top voted, non community-wiki answers of a minimum length are eligible