Questions tagged [metric]
A metric is a function that outputs a distance between 2 elements of a set & meets certain strict criteria (some 'distance' functions are not metrics).
404
questions
1
vote
1
answer
22
views
Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)
Consider these two approaches to evaluating a classifiers performance:
Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
0
votes
0
answers
14
views
Are there any research papers which show why Wasserstein distance is better than Jensen-Shannon/KL_div/Bhattacharya distance for specific use cases?
I am trying to find reliable research work which show why displacement based metrics such as Wasserstein distance is a better suited metric than Jensen-Shannon distance in specific use cases and for ...
- 31
0
votes
0
answers
24
views
How to measure the difference between two distributions of the same family?
Kullback-Leibler divergence seems to be a frequently used "metric" to measure the difference between probability distributions, regardless of their respective families. However, I would like ...
1
vote
1
answer
46
views
Finding a source for the definition of "clustering accuracy"
In papers about unsupervised clustering I see a lot of references to a metric "clustering accuracy" or "unsupervised clustering accuracy" (ACC) which is usually defined as ...
- 11
2
votes
1
answer
44
views
Test or training data? R², predicted R² and adjusted R²
I would like to understand the difference between simple R², predicted R², and adjusted R². I have done several research and readings, but the difference is still not clear to me. I have even reached ...
- 21
4
votes
2
answers
337
views
Connection between LASSO regression and Taxicab Geometry
The following is written on the Wikipedia entry of Taxicab Geometry:
The [taxicab] geometry has been used in regression analysis since the 18th century, and is often referred to as LASSO.
...
- 143
7
votes
2
answers
277
views
Measure of difference or distinctiveness that's comparable across both discrete and continuous variables
I am a product manager for a data analytics startup, and on our platform our clients end up with large tables of data about their customers which has many attributes, which are of mixed datatype: some ...
- 171
0
votes
0
answers
139
views
How to calculate FID for a set with a small number of images?
I need to evaluate my generative model using FID (Fréchet inception distance). However, the dataset of real images that I have only contains 2719 examples. I've read that the authors of the metric ...
- 101
0
votes
0
answers
67
views
Sample size calculation for AB testing - Non binomial ratio metric
I'm currently working on a sample size calculation for an upcoming AB test related to our mobile app. Up until now, I've been dealing with binomial metrics, such as the conversion rate, which is ...
- 101
0
votes
1
answer
20
views
Why do we always consider 2-variable affinity metrics?
Clustering algorithms often use some affinity metrics to cluster a dataset. Given some data points $x_1, x_2, \ldots, x_n$, it is common to compare $x_i$ and $x_j$ using a function $f(x_i, x_j)$ which ...
- 1
2
votes
1
answer
75
views
How to choose the output vector size for metric learning? [closed]
In metric learning of e.g., MNIST images, a CNN projects a 28 x 28 image into a $d$-dimensional vector which gets passed to a metric learning loss function: minimize the Euclidean or cosine distance ...
- 23
1
vote
1
answer
31
views
How can I compare two methods of measuring variables for a set of objects?
I have a set of objects for which I measured the variables in two different ways. How can I determine whether these datasets overlap well in the feature space? I was thinking about calculating some ...
0
votes
0
answers
26
views
Does this type of metric have a common name?
I tend to use metrics like the following:
"80% of all parcels are delivered within the given 60 min time period" or
"In 90% of all emergency calls, the rescue service is on the scene ...
- 101
1
vote
1
answer
95
views
Relations between the energy distance and MMD
I was wondering if there's any relation between the two metrics. Both measure the distance between distributions (or samples of them). And they seem quite similar.
The energy distance can be ...
- 3,251
2
votes
1
answer
117
views
Any good metric for measuring multi-annotator agreement on an imbalanced dataset?
Is there an agreement method that would be well-suited for a data annotation task where:
the labels are discrete classes
each datapoint belongs to exactly one class (multi-class classification)
each ...
- 43
0
votes
0
answers
52
views
Which metric to compare two probability density?
I need to compare two distribution $p$ and $q$. But I don't have access to the distribution $p$, I want to approximate it by distribution $q$ that I construct iteratively by choosing design point. ...
- 1
0
votes
0
answers
61
views
Hausdorff and IoU are stopped changing while dice metric is decreasing
I am facing a problem in Hausdorff and IoU, where they stop learning when reaching a specific value! While the loss and dice metric keeps changing. surface_distance also has a problem since it is ...
- 33
0
votes
0
answers
14
views
Distance of a time series from the white noise
Are there any ways to assign a metric to time series that measures its distance from the white noise? By white noise I mean a time series sampled from $N(0,\sigma^2)$ for some $
\sigma$. This metric ...
- 207
0
votes
0
answers
73
views
What metrics could I use to compare multiple 2-dimensional histograms/sinograms?
I have a set of two-dimensional shapes, each represented by one or more closed paths, which enclose one continuous area. My objective is to establish a measure of similarity between these shapes. I do ...
- 101
0
votes
0
answers
27
views
What are the best metrics to use for data drift detection of a Binary classification model on a tabular imbalanced numerical data
I have a very unbalanced 2 data sets, the first is the one that the model was trained on (old training data), and the second is the newly collected data
I want to see if the model performance dropped ...
1
vote
0
answers
121
views
Generalizing Inverse Variance Weighting?
Consider a generalization of inverse-variance weighting, where we choose normalized weights
$w_i = \dfrac{\sigma_i^{-p}}{W_p}$ for some $p\geq 1$, where $W_p \equiv \sum\sigma_i^{-p}$.
Then $p=2$ ...
- 111
0
votes
0
answers
21
views
Does this metric exist for clustering data based off its similarity matrix?
I'm working with a dataset where all I have is the similarity matrix (with values 0 to 1, 0 being no similarity, 1 being identical). After I assign the labels, I loop through the matrix and, for each ...
0
votes
0
answers
21
views
How can I evaluate the performance of a object detector at a fixed confidence threshold?
I have an object detector and now I have to decide which confidence threshold to use for each class. How can I determine what is the best confidence threshold for each class? Once decided, how can I ...
- 1
1
vote
0
answers
17
views
Is there a term used to refer to the total number of positive predictions?
I'm not sure how else to put it, but I often use the sklearn.metrics.classification_report function in order to measure the performance of various classification ...
- 3,466
1
vote
1
answer
39
views
Applying classification model when training and inference populations are different
I am looking for ways of estimating or mitigating the risk of applying a classification model (say logistic regression for simplicity) in a certain population (the inference set) that is known to be ...
- 11
0
votes
1
answer
68
views
Is there an error metric that decreases the weight when the target is near zero?
As precipitation prediction models can only predict positive values, they won't be able to undershoot small values by much. When it comes to overshooting, there is no boundary. High precipitation ...
2
votes
1
answer
26
views
Composed cosine similarity
I have the following problem.
I have 3 vectors $u,v,w$ of n dimensions.
I'm able to find cosine similarities between $u$ and $v$, and between $v$ and $w$: $cosine(u,v)$ and $cosine(v,w)$.
Can i use ...
- 21
0
votes
0
answers
30
views
MCC? dPrime? Which evaluation metric for the skill of detecting highly unbalanced classes
Newbie to the confusion matrix here who ended up being very confused about potential evaluation metrics.
I want to understand the detection skill that groups of human annotators possess when it comes ...
0
votes
0
answers
125
views
Why weighted F1-measure, precision and recall are always very similar to accuracy in my problem?
I'm adopting accuracy and macro and weighted averages of precision, recall and f-measure for evaluating my model in a multiclass problem with an imbalanced dataset.
However, I noticed that weighted f1,...
- 718
0
votes
1
answer
37
views
Find most unique image from pair scores of a set of Images
Long story short, I have a set of vectors for each image after training on a model. I'd like to find the most unique image from the scores generated by ...
- 1
0
votes
0
answers
121
views
How to use learning curves and cross-validation?
My aim is to prove whether there is overfitting or underfitting. However, when I calculate the learning curves (graphically depict how a process is improved), the standard deviation of the cross-...
- 13
0
votes
0
answers
15
views
Flipping inputs in multilabel classification
I have framed a classification problem as follows:
I have $N$ items, and wish to predict a set of relevant tags for each out of $M$ tags. An item can have anywhere from 0 to $M$ applicable tags.
To ...
- 1
1
vote
1
answer
37
views
Statistical argument for this cluster measure
This quick clustering score discussion presents the following single cluster scoring functions:
$$
c_i = 1-\sqrt{\frac{\sum_{j}^{N}\left(1-\phi(i,j) \right)^2}{N-1}}
$$
and
$$
C = 1-\sqrt{\frac{\sum_{...
- 123
0
votes
0
answers
40
views
From pointwise convergence to uniform: metrics
Let $\mu_\theta$ be the limit of an empirical measure $\mu_{n, \theta}$. $\theta \in \Theta$ and $\Theta$ is a compact set. Morever, the maps $\theta \rightarrow \mu_{n, \theta}$ and $\theta \...
- 309
0
votes
0
answers
12
views
Ages deviation in custom metric
I'm making custom metric for evaluation of genetic algorithm made to form teams of most appropriate people. I.e. I'd wanted to be people from one city, similar <...
- 211
0
votes
1
answer
28
views
In a regression model, what does RSS divided by predicted response signify?
I know that RSS explains the deviation between the model and actual data by measuring the square of the difference between them. I found this metric in one of the questions sir gave me. What is the ...
- 103
1
vote
1
answer
118
views
How to evaluate Natural Question-Answer Generation pairs?
I am trying to generate Natural Question-Answer for a specific domain. I am using a Large Language Model (LLM). I have only context to generate question-answers but don't have any ground truth. How to ...
- 313
0
votes
0
answers
120
views
Relationship between precision-recall curve and Fmax
These two metrics are both usually appropriate for imbalanced classification. Since
$$F_{max} = \max{\frac{2\cdot precision\cdot recall}{precision+recall}},$$
I'm guessing Fmax might be somewhat "...
- 21
1
vote
1
answer
57
views
Do you know any technique for checking if a machine learning regression model gives different performance for different values of features?
Let’s say I have fitted a regression model on a large dataset, and I obtain an estimation performance metrics (R-squared, RMSE) for the whole dataset, via cross validation or via a test set. However, ...
0
votes
1
answer
22
views
In what case is PR AUC higher than ROC AUC?
I am working on an anomaly detection problem and have come across a paper(https://www.ijcai.org/proceedings/2019/0378.pdf), which shows results where in the ROC AUC value for a dataset is 0.566 and ...
3
votes
1
answer
150
views
Singular beta in the F-beta vs. threshold score?
Consider this plot of the $F_\beta$ score for different values of $\beta$. I have a hard time getting an intuition as to why they intersect at a same point. (Cf. this blog post.) In other words, why ...
- 765
0
votes
0
answers
9
views
What is a good dissimilarity on rooted ordered labeled trees with probabilities on the leaves?
Consider a rooted ordered labeled tree that is "binary" in that there are at most two children of any vertex and whose leaves are decorated with probabilities (the sum of the leaf ...
- 141
2
votes
1
answer
160
views
Is there a binary classification metric that optimizes the count of True Positive predictions given an upper bound on False Discovery Rate?
I'm looking for a metric that behaves like this
Here, N can be interpreted as the total number of positive predictions. (Alternatively, you can interpret N as the total number of true positives. ...
- 1,854
1
vote
0
answers
49
views
How to define a distance metric between nodes of an Origin-Destination matrix?
I have an Origin-Destination matrix expressing (weekly) flows of people between every couple of nodes (cities). The number of people traveling from city $i$ to city $j$ in a specific week is $OD_{ij}$....
- 11
2
votes
1
answer
122
views
Intuitition behind Average Absolute Odds fairness
While reading a FairML paper, I saw that they use Average Absolute Odds, whose math definition will be:
$\text{FPR} = \frac{\text{FP}}{\text{FP + TN}} $
$\text{Average Absolute Odds} = |TPR_j - FPR_j| ...
- 350
0
votes
0
answers
30
views
Evaluation metrics of XAI techniques output
Consider multiple videos with N frames. I have M models and X XAI methods. Basically, i've trained and evaluated some models (classification task, real/deepfake) on these N frames and I've obtained ...
- 1
1
vote
1
answer
105
views
F1 score for change point detection
I'm trying to evaluate my change point detection algorithm based on F1 score[1] defined as follows
Let $\mathcal{X}$ denote the set of change point locations provided by a detection algorithm and let $...
- 350
0
votes
1
answer
38
views
Name and origin of a metric for evaluating ranking performance
I came across an evaluation metric to test whether a predicted rank is good, especially the top k items. But I don't know what it is called or where it is used, which makes my discussion of this ...
2
votes
1
answer
167
views
How to *formalize mathematically* that a binary classifier has no predictive performance?
The objective of supervised learning is to induce a function $f_\theta$, where $f_\theta$ is from a family of functions $f_\theta \in F$, from a training set $D^{tr}=\{(x_0^{tr},y_0^{tr})\ldots, (x_n^...
- 350
1
vote
1
answer
62
views
What is the level of measurement of my dependent variable?
For my master thesis I conducted an online experiment where participants had to conduct a shopping task where they were provided with a local and a non-local product three times in a row. So for three ...
- 71