Questions tagged [ranking]
Ranking is the task (for respondents) or the result of ordering the given stimuli or performers from "highest" to "lowest" (or *vice versa*) in some respect. It is usually contrasted with rating of stimuli. (For ranking as a way of data transformation - use tag [ranks]).
311
questions with no upvoted or accepted answers
5
votes
1
answer
294
views
Smallest possible difference between AUC of two ranker
If there are 10 positive examples, and 90 negative examples in the test set, what is the smallest possible difference in AUC, between two rankers giving different AUC?
5
votes
1
answer
307
views
Model relation between two rank variables where ranks are nested within subjects in one variable
I have elicited 10 attributes from $N$ subjects. Each subject rank ordered his own 10 attributes from the most to the least important one. I am interested in the relation between the order of ...
5
votes
0
answers
1k
views
Learning to Rank: query-dependent vs. query-independent features
I've been doing some reading about learning to rank - specifically lambdaMART - and one thing I am confused about is the role of features. When training a model, should one only use query-dependent ...
5
votes
0
answers
86
views
(Nominal) raters with no gold standard
A friend of mine took a document and broke it up into parts, then asked 5 subject matter experts to classify each part into nominal category A, B, C, D, or E. (I'm not sure yet, but D may be "All of ...
4
votes
0
answers
61
views
Rank forecasting from uneven samples
Say we have a bag of chocolate balls. There are $I$ different unique colors. Our bag has an unknown number of balls for each color.
We want to get a sense of which color people like the most and ...
4
votes
1
answer
2k
views
Friedman's test to identify best of multiple classifiers on multiple domains
I have several classifiers $f_i\ (i=1, \cdots, N)$ and calculated performance measures on multiple domains $(D)$ for each. Thus, there are $N \times D$ values.
I want to find out (increasing ...
4
votes
0
answers
1k
views
Why are ERR (Expected Reciprocal Ranking) scores not normalized?
It seems to me that normalized ERR (Expected Reciprocal Ranking) scores (ERR scores of your ranking algorithm divided by ERR score calculated for the ground truth ranking) are more useful than the ...
4
votes
0
answers
200
views
Friedman test to report on occurrence of categories in multiple articles
To investigate reasons for certain behavior, we have defined three categories for these reasons, and have determined how often these were mentioned across 15 (very different) scientific articles. ...
4
votes
0
answers
575
views
Comparing the difference between two ranks
I have a list of teams in a local soccer league, ranked from 1 to 10, where the top (1st) ranked team has the most wins (and hence "points").
I'm trying to create a "power" list, so that if the 10th ...
4
votes
1
answer
159
views
I have 12 items to get into a single rank order. Can I get this from having a few hundred people see a set of 10 of the possible pairs of items?
I attempted to run a study where people had to rank order 12 statements from best to worst. However, it was extremely messy and difficult for people to rank that many items. Now, my professor and I ...
3
votes
0
answers
151
views
How to determine sample size for number of pairwise comparisons for item ranking
Suppose I have 100 items and I would like to come up with their ranking based on user preferences. i.e. each item should receive a rank from between 1-100, where 1 is the most preferred item and 100 ...
3
votes
0
answers
33
views
Ranking data using a weight system
I've collected data from 4 people (id) on their number of violations at work regarding (1) logistics, (2) doing ...
3
votes
1
answer
281
views
Skill score/Rating systems that take into consideration margin of victory
I am trying to create ratings/skill scores using ranking systems like Elo, TrueSkill, and OpenSkill. All of these systems are similar in that they have a distribution that represents players' skill ...
3
votes
0
answers
47
views
asking humans to rank items
I have around 50 items and I need to ask human graders to rank them. Is there any good resource on how to design this crowd sourcing task? For example, it will be tiresome to ask humans to rank 50 ...
3
votes
1
answer
53
views
Statistically Comparing (Heterogeneous) Preferences
Are there any statistical methods that can compare (heterogeneous) preferences?
For instance, suppose there are two people (Person 1 and Person 2). Both of these people make a list of their favorite ...
3
votes
0
answers
40
views
Is Average Precision for ranking problems computed the same way as in classification?
For classification, we calculate Average Precision (AP) as area under the PR curve. But for ranking problems, we calculate it via the relevance of the items at certain ranked positions. Are these 2 ...
3
votes
0
answers
44
views
How to control the error rate when selecting the mode of a sample as the mode of the population?
I just looked out of my window and saw 4 white, 2 black, 1 green, 1 silver, 1 light blue, 1 lemon and 1 orange-white-striped car. Based on this, how confident should I be that white is the most common ...
3
votes
0
answers
734
views
What is exactly query group "qid" in XGBoost
In XGBoost documentation it's said that for ranking applications we can specify query group ID's qid in the training dataset as in the following snippet:
...
3
votes
0
answers
155
views
When should one use Bradley-Terry instead of gradient boosted trees for pairwise ranking
Both the Bradley-Terry model and Gradient boosted trees can be used to learn a ranking from pairwise comparisons (e.g. with libraries choix and XGboost).
How do they relate to each other? Is there ...
3
votes
0
answers
21
views
Options for nonparametric preference ranking models
What are some options for nonparametric ranking models with a probabilistic interpretation?
I'm basically looking for a nonlinear/nonparametric version of a rank ordered logit model.
I'm aware of ...
3
votes
0
answers
69
views
non-parametric circular statistics
I am working on a research question of color preference (which can be represented in a circular space).
I have divided the circular space into 16 equally spaced bins and done a pairwise comparison ...
3
votes
0
answers
37
views
Comparing a predicted order of events to actual order of events
Sorry about my somewhat limited knowledge of statistics, it's just this is a tad too complex for my knowledge of the subject.
I'm running a competition where entrants are told what events happen, and ...
3
votes
0
answers
75
views
Why does rank correlation only depend on copula, and not on the margins, if the margins are continuous
I read that for continuous margins, then Rank correlation only depend on copula and not on the margins. However, it is not the case for the non-continuous margins. Is that because the Kendall's tau (...
3
votes
1
answer
643
views
Modelling Races
The question that I wish to ask is what is an appropriate likelihood model to use for races.
For example, suppose that we have 4 competitors in a 100m race. We observe the competitors weight and age ...
3
votes
0
answers
418
views
Argmax order statistics for multivariate normal
The Problem
Given a gaussian random variable $\mathbf{x} = (x_1, ..., x_N)^T \sim \mathcal{N}(0, \Sigma)$ what is the probability that $i = \underset{i\in\{1, \dots, N\}}{argmax}\{|x_1|, \dots, |x_N|\...
3
votes
1
answer
45
views
Ranking highest scoring sample across different assays
So I have a set of data that is composed from 7 samples tested across different assays. In each assay there is an order of data from the highest scoring sample to the lowest scoring one.
I would like ...
3
votes
0
answers
52
views
Valuation of players across uneven games
I have a 'portfolio' of players that I need to evaluate and compare across many games. There is a common metric,let's call it value, I am using to evaluate a players performance in a single game.
...
3
votes
0
answers
581
views
Is it appropriate to calculate the average value of a correlation matrix?
I have several sets of ranking lists made by 14 judges. Each set represents a number of videos that have been ordered with respect to a particular characteristic. Moreover, the videos are the same for ...
3
votes
0
answers
211
views
Creating a scoring/ranking variable
I have been tasked with creating a variable which is a score. There are roughly 5-10 variables which I will be factoring into this score variable. I have looked into dimensionality reduction as a ...
3
votes
0
answers
778
views
Merge Multiple Rankings (different criteria and observations) into one
So I have 3 rankings on different sets of companies (different industries) and each of them was elaborated based on the key performance indicators of that particular industry.
I ran a PCA to find the ...
3
votes
0
answers
393
views
What are some alternatives to the random surfer model in the Page Rank algorithm?
In the original PageRank algorithm, the authors introduced the idea of a random surfer,who was a user who while surfing the web, could either go from the current web page to a web page by clicking on ...
3
votes
0
answers
75
views
Can rank ordered mixed logit handle varying set sizes?
I have data that consist of numerous trials, where each trial is a ranked set of items. The set of items for each trial is a subset of varying size of all possible items. (I.e. the items are shared ...
3
votes
0
answers
121
views
Evaluation of a semi-supervised ranking model
I learned ranking model with graph-based semi-supervised approach, while labeled (just positive) and unlabeled (positive and negative) data is both used in training. With the model, all of the data ...
3
votes
0
answers
267
views
Comparing Elo Ranking system with another ranking system
I'm a guy who is interested in computer science and algorithms. I've spent a bit of time recently developing my own ranking system based on PageRank.
I'd like to compare it to the Elo system, in ...
3
votes
0
answers
164
views
Google consumer survey stats
In Google's consumer survey product, they calculate confidence intervals using the Wilson score interval for the unweighted results.
Does anyone know how they calculate the confidence interval after ...
3
votes
0
answers
622
views
Handling zero scores in "weighted product model"?
I am conducting analysis on 'Multi-Attribute Decision Making (MADM)', where I have two attributes (a1, a2) to characterize the quality of m alternative approaches. The first attribute, a1, is measured ...
3
votes
0
answers
472
views
How to modify RankBoost to maximize area under recall-precision curve instead of AUC?
Using the WeakLearn algorithm from the original RankBoost paper, how do you set the optimal threshold to maximize AU-RPC (instead of AUC)? And, once that threshold is set, how do you calculate the AU-...
3
votes
1
answer
170
views
What's the correct method to rank populations?
I want to rank ten or so sources based on their respective error measurements.
A few notes on the sources and measurements:
The sources do not have the same number of measurements
The number of ...
2
votes
0
answers
38
views
rank aggregation with partial ranking lists
My question is about rank aggregation, which is in relation with statistics and decision sciences. (if this question is off the topic of this site, plz let me know.) The question is that I have N ...
2
votes
0
answers
32
views
Best way of determining how unique an inspector's ratings are compared to the average inspector
Noob question.
I have a full population of every inspector rating for all items being rated. Each inspector has inspected a subset of the items and given them a rating, but different inspectors have ...
2
votes
1
answer
87
views
Optimizing XGBClassifier for discrimination power
I am developing a credit risk decisioning model, i.e. a model that assesses the risk of default of an incoming transactions and decides whether to accept it or not. Of course my dataset is imbalanced :...
2
votes
0
answers
118
views
What is the best way to combine individual rankings to provide an overall group ranking?
If I have 50 people randomly split into 5 groups of 10 people, I can rank the groups based on their combined weight. If I were to rank each person based on their individual weight, what would be the ...
2
votes
0
answers
14
views
I need to create a neural network architecture for learning to rank task
I need ranking comments according to the text. They are represented by a vector representation from Sentence transformers. One text and 5 comments, and a score from 0 to 4 for each comment.
I need ...
2
votes
0
answers
37
views
Comparing the similarity of ranked word lists with a reference standard
Imagine a scientific organisation wants to communicate the likelihood of particular events occurring. For consistency, the organisation decides to always draw from the same set of words, which it ...
2
votes
0
answers
209
views
Different Number of Judges Scoring and how to Rank
I have looked everywhere and I am struggling to find a solution. Granted, I am no statistician but I do have some knowledge of basic statistics. Any help is truly appreciated!
There are a number of ...
2
votes
1
answer
135
views
Linear Regression - Data Subset with Lowest Mean-Squared Error?
Short version: given a linear regression dataset and an integer $K$, what data subset of size $K$ results in linear parameters with the lowest mean squared error on the entire dataset?
Long version: ...
2
votes
0
answers
18
views
terms used in Stochastic ordering
I have some rudimentary questions on how to call/name certain things in terms of Stochastic ordering. I am using simple examples here but would like to know the language used in general cases.
...
2
votes
0
answers
30
views
Testing tuples of response variables
I'm struggling with comparing samples with multiple response variables, where such variables have a strict precedence or importance order. For the sake of this exercise, let's assume that each sample ...
2
votes
0
answers
11
views
Nonparametric estimation of ranking data
Suppose I have $\{X_i, Y_i, N_i\}$, where $N_i$ is the number of objects for sample $i$ (typically 30 - 35 objects), $Y_i$ is complete ranking (ordering or permutation) of $N_i$ objects, and $X_i = (...
2
votes
0
answers
37
views
Rank-ordered statistics of a data time series
I have a time series of observations $(x_t)_{1\leq t\leq T}$ that I can order into $(x_n)_{1\leq n\leq N}$, with $N = T$ such that $x_n<x_{n+1}$ and $x_1=\min{x_t}$ and $x_N=\max{x_t}$.
Then I pair ...