Questions tagged [ranking]

Ranking is the task (for respondents) or the result of ordering the given stimuli or performers from "highest" to "lowest" (or *vice versa*) in some respect. It is usually contrasted with rating of stimuli. (For ranking as a way of data transformation - use tag [ranks]).

311 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5 votes
1 answer
294 views

Smallest possible difference between AUC of two ranker

If there are 10 positive examples, and 90 negative examples in the test set, what is the smallest possible difference in AUC, between two rankers giving different AUC?
wrek's user avatar
  • 185
5 votes
1 answer
307 views

Model relation between two rank variables where ranks are nested within subjects in one variable

I have elicited 10 attributes from $N$ subjects. Each subject rank ordered his own 10 attributes from the most to the least important one. I am interested in the relation between the order of ...
Mark Heckmann's user avatar
5 votes
0 answers
1k views

Learning to Rank: query-dependent vs. query-independent features

I've been doing some reading about learning to rank - specifically lambdaMART - and one thing I am confused about is the role of features. When training a model, should one only use query-dependent ...
keeno's user avatar
  • 51
5 votes
0 answers
86 views

(Nominal) raters with no gold standard

A friend of mine took a document and broke it up into parts, then asked 5 subject matter experts to classify each part into nominal category A, B, C, D, or E. (I'm not sure yet, but D may be "All of ...
Wayne's user avatar
  • 21.1k
4 votes
0 answers
61 views

Rank forecasting from uneven samples

Say we have a bag of chocolate balls. There are $I$ different unique colors. Our bag has an unknown number of balls for each color. We want to get a sense of which color people like the most and ...
Amelio Vazquez-Reina's user avatar
4 votes
1 answer
2k views

Friedman's test to identify best of multiple classifiers on multiple domains

I have several classifiers $f_i\ (i=1, \cdots, N)$ and calculated performance measures on multiple domains $(D)$ for each. Thus, there are $N \times D$ values. I want to find out (increasing ...
Chris's user avatar
  • 599
4 votes
0 answers
1k views

Why are ERR (Expected Reciprocal Ranking) scores not normalized?

It seems to me that normalized ERR (Expected Reciprocal Ranking) scores (ERR scores of your ranking algorithm divided by ERR score calculated for the ground truth ranking) are more useful than the ...
AatG's user avatar
  • 181
4 votes
0 answers
200 views

Friedman test to report on occurrence of categories in multiple articles

To investigate reasons for certain behavior, we have defined three categories for these reasons, and have determined how often these were mentioned across 15 (very different) scientific articles. ...
Derk Arts's user avatar
  • 131
4 votes
0 answers
575 views

Comparing the difference between two ranks

I have a list of teams in a local soccer league, ranked from 1 to 10, where the top (1st) ranked team has the most wins (and hence "points"). I'm trying to create a "power" list, so that if the 10th ...
Sam's user avatar
  • 141
4 votes
1 answer
159 views

I have 12 items to get into a single rank order. Can I get this from having a few hundred people see a set of 10 of the possible pairs of items?

I attempted to run a study where people had to rank order 12 statements from best to worst. However, it was extremely messy and difficult for people to rank that many items. Now, my professor and I ...
Bofstein's user avatar
3 votes
0 answers
151 views

How to determine sample size for number of pairwise comparisons for item ranking

Suppose I have 100 items and I would like to come up with their ranking based on user preferences. i.e. each item should receive a rank from between 1-100, where 1 is the most preferred item and 100 ...
phil's user avatar
  • 341
3 votes
0 answers
33 views

Ranking data using a weight system

I've collected data from 4 people (id) on their number of violations at work regarding (1) logistics, (2) doing ...
rnorouzian's user avatar
  • 3,986
3 votes
1 answer
281 views

Skill score/Rating systems that take into consideration margin of victory

I am trying to create ratings/skill scores using ranking systems like Elo, TrueSkill, and OpenSkill. All of these systems are similar in that they have a distribution that represents players' skill ...
Jage's user avatar
  • 173
3 votes
0 answers
47 views

asking humans to rank items

I have around 50 items and I need to ask human graders to rank them. Is there any good resource on how to design this crowd sourcing task? For example, it will be tiresome to ask humans to rank 50 ...
A.M.'s user avatar
  • 131
3 votes
1 answer
53 views

Statistically Comparing (Heterogeneous) Preferences

Are there any statistical methods that can compare (heterogeneous) preferences? For instance, suppose there are two people (Person 1 and Person 2). Both of these people make a list of their favorite ...
stats_noob's user avatar
3 votes
0 answers
40 views

Is Average Precision for ranking problems computed the same way as in classification?

For classification, we calculate Average Precision (AP) as area under the PR curve. But for ranking problems, we calculate it via the relevance of the items at certain ranked positions. Are these 2 ...
oldselflearner1959's user avatar
3 votes
0 answers
44 views

How to control the error rate when selecting the mode of a sample as the mode of the population?

I just looked out of my window and saw 4 white, 2 black, 1 green, 1 silver, 1 light blue, 1 lemon and 1 orange-white-striped car. Based on this, how confident should I be that white is the most common ...
jan-glx's user avatar
  • 361
3 votes
0 answers
734 views

What is exactly query group "qid" in XGBoost

In XGBoost documentation it's said that for ranking applications we can specify query group ID's qid in the training dataset as in the following snippet: ...
Konstantin's user avatar
3 votes
0 answers
155 views

When should one use Bradley-Terry instead of gradient boosted trees for pairwise ranking

Both the Bradley-Terry model and Gradient boosted trees can be used to learn a ranking from pairwise comparisons (e.g. with libraries choix and XGboost). How do they relate to each other? Is there ...
vman's user avatar
  • 173
3 votes
0 answers
21 views

Options for nonparametric preference ranking models

What are some options for nonparametric ranking models with a probabilistic interpretation? I'm basically looking for a nonlinear/nonparametric version of a rank ordered logit model. I'm aware of ...
Thomas Johnson's user avatar
3 votes
0 answers
69 views

non-parametric circular statistics

I am working on a research question of color preference (which can be represented in a circular space). I have divided the circular space into 16 equally spaced bins and done a pairwise comparison ...
ha554an's user avatar
  • 83
3 votes
0 answers
37 views

Comparing a predicted order of events to actual order of events

Sorry about my somewhat limited knowledge of statistics, it's just this is a tad too complex for my knowledge of the subject. I'm running a competition where entrants are told what events happen, and ...
Chris Johnson's user avatar
3 votes
0 answers
75 views

Why does rank correlation only depend on copula, and not on the margins, if the margins are continuous

I read that for continuous margins, then Rank correlation only depend on copula and not on the margins. However, it is not the case for the non-continuous margins. Is that because the Kendall's tau (...
F.Thomas's user avatar
  • 138
3 votes
1 answer
643 views

Modelling Races

The question that I wish to ask is what is an appropriate likelihood model to use for races. For example, suppose that we have 4 competitors in a 100m race. We observe the competitors weight and age ...
sachinruk's user avatar
  • 1,353
3 votes
0 answers
418 views

Argmax order statistics for multivariate normal

The Problem Given a gaussian random variable $\mathbf{x} = (x_1, ..., x_N)^T \sim \mathcal{N}(0, \Sigma)$ what is the probability that $i = \underset{i\in\{1, \dots, N\}}{argmax}\{|x_1|, \dots, |x_N|\...
jds's user avatar
  • 978
3 votes
1 answer
45 views

Ranking highest scoring sample across different assays

So I have a set of data that is composed from 7 samples tested across different assays. In each assay there is an order of data from the highest scoring sample to the lowest scoring one. I would like ...
Alan_dj's user avatar
  • 69
3 votes
0 answers
52 views

Valuation of players across uneven games

I have a 'portfolio' of players that I need to evaluate and compare across many games. There is a common metric,let's call it value, I am using to evaluate a players performance in a single game. ...
bzak's user avatar
  • 143
3 votes
0 answers
581 views

Is it appropriate to calculate the average value of a correlation matrix?

I have several sets of ranking lists made by 14 judges. Each set represents a number of videos that have been ordered with respect to a particular characteristic. Moreover, the videos are the same for ...
TobsenB's user avatar
  • 131
3 votes
0 answers
211 views

Creating a scoring/ranking variable

I have been tasked with creating a variable which is a score. There are roughly 5-10 variables which I will be factoring into this score variable. I have looked into dimensionality reduction as a ...
gold_cy's user avatar
  • 173
3 votes
0 answers
778 views

Merge Multiple Rankings (different criteria and observations) into one

So I have 3 rankings on different sets of companies (different industries) and each of them was elaborated based on the key performance indicators of that particular industry. I ran a PCA to find the ...
Orwell's user avatar
  • 31
3 votes
0 answers
393 views

What are some alternatives to the random surfer model in the Page Rank algorithm?

In the original PageRank algorithm, the authors introduced the idea of a random surfer,who was a user who while surfing the web, could either go from the current web page to a web page by clicking on ...
user3676846's user avatar
3 votes
0 answers
75 views

Can rank ordered mixed logit handle varying set sizes?

I have data that consist of numerous trials, where each trial is a ranked set of items. The set of items for each trial is a subset of varying size of all possible items. (I.e. the items are shared ...
uberpro's user avatar
  • 41
3 votes
0 answers
121 views

Evaluation of a semi-supervised ranking model

I learned ranking model with graph-based semi-supervised approach, while labeled (just positive) and unlabeled (positive and negative) data is both used in training. With the model, all of the data ...
xueliang liu's user avatar
3 votes
0 answers
267 views

Comparing Elo Ranking system with another ranking system

I'm a guy who is interested in computer science and algorithms. I've spent a bit of time recently developing my own ranking system based on PageRank. I'd like to compare it to the Elo system, in ...
Peter's user avatar
  • 31
3 votes
0 answers
164 views

Google consumer survey stats

In Google's consumer survey product, they calculate confidence intervals using the Wilson score interval for the unweighted results. Does anyone know how they calculate the confidence interval after ...
ppham27's user avatar
  • 131
3 votes
0 answers
622 views

Handling zero scores in "weighted product model"?

I am conducting analysis on 'Multi-Attribute Decision Making (MADM)', where I have two attributes (a1, a2) to characterize the quality of m alternative approaches. The first attribute, a1, is measured ...
banbar's user avatar
  • 131
3 votes
0 answers
472 views

How to modify RankBoost to maximize area under recall-precision curve instead of AUC?

Using the WeakLearn algorithm from the original RankBoost paper, how do you set the optimal threshold to maximize AU-RPC (instead of AUC)? And, once that threshold is set, how do you calculate the AU-...
Travis's user avatar
  • 31
3 votes
1 answer
170 views

What's the correct method to rank populations?

I want to rank ten or so sources based on their respective error measurements. A few notes on the sources and measurements: The sources do not have the same number of measurements The number of ...
c00kiemonster's user avatar
2 votes
0 answers
38 views

rank aggregation with partial ranking lists

My question is about rank aggregation, which is in relation with statistics and decision sciences. (if this question is off the topic of this site, plz let me know.) The question is that I have N ...
Maul Seil's user avatar
2 votes
0 answers
32 views

Best way of determining how unique an inspector's ratings are compared to the average inspector

Noob question. I have a full population of every inspector rating for all items being rated. Each inspector has inspected a subset of the items and given them a rating, but different inspectors have ...
inspecting_inspectors's user avatar
2 votes
1 answer
87 views

Optimizing XGBClassifier for discrimination power

I am developing a credit risk decisioning model, i.e. a model that assesses the risk of default of an incoming transactions and decides whether to accept it or not. Of course my dataset is imbalanced :...
Anatole's user avatar
  • 21
2 votes
0 answers
118 views

What is the best way to combine individual rankings to provide an overall group ranking?

If I have 50 people randomly split into 5 groups of 10 people, I can rank the groups based on their combined weight. If I were to rank each person based on their individual weight, what would be the ...
user384069's user avatar
2 votes
0 answers
14 views

I need to create a neural network architecture for learning to rank task

I need ranking comments according to the text. They are represented by a vector representation from Sentence transformers. One text and 5 comments, and a score from 0 to 4 for each comment. I need ...
Dmitry  Sokolov's user avatar
2 votes
0 answers
37 views

Comparing the similarity of ranked word lists with a reference standard

Imagine a scientific organisation wants to communicate the likelihood of particular events occurring. For consistency, the organisation decides to always draw from the same set of words, which it ...
MoonResearch's user avatar
2 votes
0 answers
209 views

Different Number of Judges Scoring and how to Rank

I have looked everywhere and I am struggling to find a solution. Granted, I am no statistician but I do have some knowledge of basic statistics. Any help is truly appreciated! There are a number of ...
Marie's user avatar
  • 21
2 votes
1 answer
135 views

Linear Regression - Data Subset with Lowest Mean-Squared Error?

Short version: given a linear regression dataset and an integer $K$, what data subset of size $K$ results in linear parameters with the lowest mean squared error on the entire dataset? Long version: ...
Rylan Schaeffer's user avatar
2 votes
0 answers
18 views

terms used in Stochastic ordering

I have some rudimentary questions on how to call/name certain things in terms of Stochastic ordering. I am using simple examples here but would like to know the language used in general cases. ...
Adam's user avatar
  • 21
2 votes
0 answers
30 views

Testing tuples of response variables

I'm struggling with comparing samples with multiple response variables, where such variables have a strict precedence or importance order. For the sake of this exercise, let's assume that each sample ...
an_drade's user avatar
  • 121
2 votes
0 answers
11 views

Nonparametric estimation of ranking data

Suppose I have $\{X_i, Y_i, N_i\}$, where $N_i$ is the number of objects for sample $i$ (typically 30 - 35 objects), $Y_i$ is complete ranking (ordering or permutation) of $N_i$ objects, and $X_i = (...
user1292919's user avatar
2 votes
0 answers
37 views

Rank-ordered statistics of a data time series

I have a time series of observations $(x_t)_{1\leq t\leq T}$ that I can order into $(x_n)_{1\leq n\leq N}$, with $N = T$ such that $x_n<x_{n+1}$ and $x_1=\min{x_t}$ and $x_N=\max{x_t}$. Then I pair ...
Douar Gwenn's user avatar

1
2 3 4 5
7