Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Ranking is ordering data from highest to lowest or *vice versa.* For questions about *constructing* scores to use in ranking, please use the [rating] tag, too.

0
votes
0answers
14 views

What is the name of this test for the probability of differences between lists and is it valid?

And are there any better methods? In a 1969 book about investing, the author describes a method for assessing the effectiveness of a stock picker. Ask the picker to provide a list of n stocks that ...
3
votes
2answers
56 views

Is ratio of ranks a well-studied statistic?

I have two lists of recommendations: that is, two different algorithms have assigned ranks to the same list of objects. I would like to know if they're similar. I specifically care more about ...
0
votes
0answers
6 views

Fitting heteroscedastic noisy rank data

I am trying to fit a model from a continuous independent variable to a rank that starts from 0 and increases. What the rank indicates and what the independent variable is, is beyond the scope of this ...
0
votes
0answers
7 views

Expected average precision for random ranked list

Suppose we are evaluating the label ranking average precision (LRAP), which is an evaluation metric used to compare a ranked list to a ground truth (binary) label vector. Background and Definition ...
1
vote
0answers
15 views

Using Class Probabilities from Random Forest to do Ranking

Assume that we have set of documents which are being labelled as relevant and non-relevant. Can I used the class probabilities generated by Random Forest in python scikit-learn to rank the documents ...
0
votes
0answers
9 views

Random Forest in RankLib

I am building a LETOR model using Random Forest with RankLib. As we know that the Random Forest gets better with higher number of trees, I got a totally different observation in my case. The ...
0
votes
1answer
15 views

Robustness of ranking based on cross validation set, should I bootstrap?

I have a dataset that I separate into training and validation. I fit several models on the training set and then evaluate their performance on the validation set. When I rank the models based on their ...
0
votes
0answers
11 views

Rank Test for a Matrix

Suppose I have a matrix A corrupted with noise and I am looking for a test that tests the null hypothesis that the matrix A, rank(A)==1 v.s. rank(A)>1. I checked a little the literature and this paper ...
0
votes
0answers
10 views

Directly optimizing over AUC surrogate

This is in relation to this paper I am looking for ways to optimize Recall @ fixed Precision ($R@P$) for a machine learning problem and i didnt want to use accuracy as a proxy for $R@P$. Upon ...
0
votes
0answers
8 views

Somers' D deteriorating for more granular rankings

Background I have 30 observations and I want to compare how much experts and model agree with each other. For Model and Expert I have slots which are $\in [1,4]$ and ranking which are $\in [1,30]$. ...
1
vote
1answer
38 views

Assigning probabilities to processes, when you know which probabilities exist but not which one belongs to which process

I have multiple stochastic processes that can have 2 different realisations. I think of them as coins that can take the values heads or tails. For the example I assume that there are 3 coins. I know ...
1
vote
0answers
26 views

Merging three rankings about economic relations

I am facing a -probably- simple or stupid problem, however I don't have the prequisite knowledge. I would like to make an overall ranking, to compare bilateral economic relations (during a given time-...
2
votes
0answers
22 views

Skill-based ranking system with individual performance

I am attempting to rank players in an online game and have successfully used TrueSkill to create a good ranking system however I would like to take it a step further and let individual performance ...
0
votes
0answers
15 views

Scaling/weighting probabilities

I have data which looks like the table below. I am provided with the likelihood of a game to occur (Prob) and the current time in min for that game (Curr_time_min). I would like to create a formula ...
0
votes
1answer
20 views

Ranking Order Test

I am analyzing my capstone data - I had students, from different grades, rate 6 feedback forms from 1-6, where 1 was the most helpful feedback type and 6 the least helpful feedback type. Is there a ...
0
votes
0answers
11 views

How are ranks defined in Page's trend test?

I'm confused about the two kinds of rank in a Page test with $n$ conditions and $k$ subjects. Wikipedia says the $i$th condition receives a rank $Y_i$ and a sum $X_i$ over the subjects' ranks. In ...
0
votes
0answers
18 views

Statistical Response for Rankings when Degree of Difference Matters

Sorry if this question has been addressed recently. I've done some searching around, but have been unable to tell if I am on the right track do to my lack of knowledge on vocabulary in the field. I ...
1
vote
0answers
20 views

What is the expected fraction of observations in the top x% that remains in the top x% after a random shock?

I'm struggling with a probability question, and I was hoping someone here could help me out. Here's the setting. Suppose you draw observations from a probability distribution Z and sort these ...
1
vote
1answer
18 views

Score ordinal ranking configuration according to an optimal ranking

I have a list of items, each associated with their optimal ranking in the list. How can I give a score to a given order of the list. For instance, if I know that the optimal ranking is: 4 3 5 1 2 And ...
0
votes
0answers
6 views

Scoring entities according to weight of its individual metrics

An individual has three metrics (each out of 100). An average of the three scores can serve as the score of the individual out of 100. See table below. However certain metrics carry a little ...
1
vote
0answers
31 views

Elo-type ranking system that incorporates game score

The Elo rating system is used to calculate relative skill levels between individuals or teams. It can be applied to many types of games and sports, but when applied, it only considers wins and losses. ...
0
votes
0answers
14 views

Score to assess ranking of predicted outcomes

Is there any quantitative measure to evaluate a predictive model's ability rank units correctly. For example, I am trying to predict a continuous outcome. The error of my predicted versus actual value ...
0
votes
0answers
16 views

Pairwise comparison for differences between Coefficient of Variation Ranking between groups

I would like to test whether the ranking obtained by ordering the coefficient of variation of some genes are different or not between two groups, and do it pairwise, so, row by row, and get a p-value ...
0
votes
0answers
27 views

What comparative analysis can I use for rank ordered data, divided into two groups, from a survey questionnaire?

I have a dataset from a survey questionnaire. Let's say I can divide the participants into group A and B based on a screening question. These participants then rank 7 different options in order from ...
0
votes
0answers
7 views

Measuring the cost of the prediction error when picking the maximum value in a sequence

I have several entities denoted as a set $A=\{1, \ldots, K\}$. They form a sequence $(y_n)_{n\in A}$ with $y_i\in \mathbb{R}$. The values of $y_i$ are unobservable and get estimated as $\hat{y}_i$. ...
4
votes
1answer
40 views

Is there a way to estimate the minimal number of pairwise comparisons required to place complete ordering over a set of items?

I have 200 items that I need users to compare. If there are 2 items, A and B, then the comparison is A before B or B before A. ...
0
votes
0answers
23 views

Compare two set

I want to compare two sets of values with a ranking. I want to have more of a similarity coefficient. My goal is to take into account both aspects: the similarity on the ranking and the similarity on ...
1
vote
0answers
18 views

Ranking items through votes

I asked participants to rank 10 items based on order of importance (i.e. 1=most important 10=least important 11=not relevant Now, I want to order the items from most voted important to least ...
0
votes
0answers
22 views

Comparing different ranking methods' results

I have been looking for literature about comparing the retrieval results of two methods. One method results in a list of ranks whereas the second method results in a list of sets of ranks. Do you have ...
0
votes
0answers
56 views

Internal consistency

I have to analyse a dataset in which there are drugs and used animals based on article review. The datasets looks like this: ...
1
vote
0answers
17 views

Making sense of AICc ranking vs. R2 and standardized coefficients

Mixed/hierarchical models and model ranking is fairly new to me. I'm just wondering if someone can help me make a little more sense out of this.... I used the lmer() to build 7 linear mixed effects ...
0
votes
0answers
8 views

Computing score ( relative score) for % Values distribution : Need review

I have a survey % values in the following order ...
1
vote
0answers
26 views

Get final rank from ranked list [duplicate]

i got exactly same problem with this question Overall rank from multiple ranked lists but i didnt get any references and answer for that. example : ...
0
votes
0answers
6 views

Aggregate main criteria and sub-criteria in AHP

When using AHP, there are main criteria and sub-criteria. One of the important step is to create the Criteria-alternative decision matrix. In case there are 6 main criteria and 20 sub-criteria. How ...
1
vote
0answers
31 views

At which rank should I reasonably stop selecting interview candidates?

It seems that my problem could be classified as an "optimal stopping" problem, but I am unable to make much of this information. It is an important problem for my organisation (an NGO), in order to ...
0
votes
1answer
32 views

Assigning weights to variables to calculate rank/score of an Agent

I have data on Agents behavior history. I want to score each of these Agents based on the attributes. Attributes are both Categorical and Continuous.For this, I want to calculate the score by ...
3
votes
3answers
231 views

Distribution of the difference between two predictions made by the same model

I have a database of 200 trees grown in different conditions, since it's very expensive to get more points I want to predict with a machine learning model the growth potential of tree seedlings: X ...
0
votes
0answers
12 views

Error of solution ranking

I am exploring subset selection of multi-armed bandits and I am curious about an error measure of a selection. So in the scenario of a multi-armed bandit, a set of arms are present and we aim to ...
0
votes
0answers
19 views

Calculating a scoring system with independent, differently weighted variables

I am working on a problem and have a solution but would like to get some feedback from you. Perhaps I am wrong.... So I am working on a scoring system in which I have five independent variables. ...
0
votes
0answers
18 views

Apply transform before or after bootstrap

I am applying some models to data. The models require that I transform the individual observations in the data by: $$ \hat{\bf{U}}_i = \frac{1}{n+1}(R_{i1}, \dots, R_{id}), \qquad i \in \{0, \dots, n\...
0
votes
0answers
43 views

Supervised vs unsupervised for anomaly detection

My problem is detecting the good/bad car reparations given the repair cost. And then I can rank the car garages based on the percentage of 'good' repair he got. A first simple model is using ...
3
votes
1answer
60 views

best-worst ranking: do all items have to be blocked together at least 1 time?

Here there are instructions to build Max-diff (or Best-worst) ranking questionnaire: In an ideal world, a MaxDiff experimental design has the following characteristics, where each alternative ...
1
vote
0answers
297 views

Proper way to use NDCG@k score for recommendations

Currently I am building a recommender system and using ranking metrics to verify its performance. I am using the NDCG@k score. Today I was experimenting and I realized that I might be calculating the ...
0
votes
0answers
17 views

Compare ranks between sets of rankings

I have a set of ~1300 genes ranked ordinally for 6 experiments (2 treatments with 3 replicates each). That is I have six sets of rankings from 1 to 1300. The replicates are not necessarily identical ...
0
votes
0answers
25 views

Approximate total ratings from groupwise ratings

I'm balancing an online game. In a particular game feature, there are a total of 20 different items, and a subset of six of these items is shown to the players each week, and they can choose some of ...
1
vote
1answer
35 views

Will normalization provide better scores?

I'm trying to come up with a decent way of providing suggestions for folders based on documents in a search result. A folder should receive a high score if all of its documents appear in a search ...
2
votes
1answer
29 views

How to decide the number of options to give to people in a poll? (for opinion based, subjective contexts)

Just to be clear, this question is open ended on purpose, I am fine with(or even prefer to) being pointed to a few research papers, and proper areas related to this subject. I am not sure what I am ...
0
votes
0answers
33 views

Feature normalization for machine-learned ranking

I am interested in trying out different machine-learned ranking algorithm in order to decrease the running time of an algorithm. I am however uncertain about feature normalization: First of all, ...
1
vote
0answers
19 views

ML Approach for Optimizing Field Boosts for Search (Information Retrieval)

I have been experimenting with different methods for tuning our search engine's field boosts. In Solr or Elastic Search, you specify the importance of matches in each field when configuring the search ...
1
vote
0answers
16 views

Predicting winner based on pairwise comparison of players

I have a problem where I have to predict the winner of a game. There could upto 10 players in each game and there is always one and only one winner. Each player also gets a rank in the game (the ...