Skip to main content

Questions tagged [metric]

A metric is a function that outputs a distance between 2 elements of a set & meets certain strict criteria (some 'distance' functions are not metrics).

Filter by
Sorted by
Tagged with
4 votes
3 answers
494 views

Judging a model through the TP, TN, FP, and FN values

I am evaluating a model that predicts the existence or not existence of a "characteristic" (for example, "there is a dog in this image") using several datasets. The system outputs ...
KansaiRobot's user avatar
1 vote
1 answer
18 views

Why RandomForestRegressor.score() return a coefficient of determination? [duplicate]

In ScikitLearn's method RandomForestRegressor.score(X, y), the coefficient of determination R_2 is returned as a metric of the ...
Maxime Charrière's user avatar
0 votes
0 answers
20 views

Kfold cross val in Regression model

How to use K-fold CV to evaluate my regression model performance to calculate the R2, MAE and MSE in the train set to make the model more robust? This code below refers to the tuned model and I'm ...
Vinicius Maia's user avatar
0 votes
0 answers
11 views

"ROC AUC reflects the likelihood that a random positive instance will be located to the right of a random negative instance". How come? [duplicate]

According to this webpage, ROC AUC reflects the likelihood that a random positive (red) instance will be located to the right of a random negative (gray) instance. Would you please explain this ...
Evan Aad's user avatar
  • 1,433
1 vote
0 answers
28 views

Why do we use both Kullback–Leibler divergence vs FID as metric for evaluating distribution distance?

Kullback–Leibler divergence is used in VAE objective function, while FID measures the quality of images of generative models. But they both measure distribution distance. Why do we have them as ...
MathematicsBeginner's user avatar
0 votes
0 answers
27 views

Survival analysis: Use of "legible" metrics like RMSE

I apply a Cox proportional hazards model to some machine failure data and I want to know, "how good" my model is. Metrics like RMSE or MAE are said to be not suitable for this kind of model, ...
Requin's user avatar
  • 621
0 votes
0 answers
19 views

Potential evaluation based on the coherence of predicted value with actual data

I have the following data over time: that means data collected for a single variable like CPU usage in lowest, highest, and average mode over time every 5 mins (data granularity = 5mins) like the ...
Mario's user avatar
  • 441
2 votes
1 answer
49 views

On using the loss as a metric?

The context is model evaluation in supervised learning. I am coming from a numerical optimisation background. For me it is quite natural to use the loss of the model (what we optimise during training) ...
Lucas Morin's user avatar
  • 1,665
0 votes
0 answers
15 views

Metric for Bayesian ground truth recovery

I am developing a new Bayesian model and want to compare it to already existing Bayesian models with the same hyperparameters using a simulation study. I generated 50 datasets and fit 4 different ...
Marcello Zago's user avatar
1 vote
2 answers
124 views

Proof of property of Mallows/Wasserstein metric

Let $\mathcal F_p=\{H\text{ is a cumulative distribution function}:\int|x|^pdH<\infty\}$. Define on $\mathcal F_p,$ Mallows' metric ($p$ Wasserstein metric), $d_p,p\ge1$ for two random variables $X,...
reyna's user avatar
  • 385
2 votes
0 answers
88 views

Maximum Mean Discrepancy (MMD) implementation as a metric to measure GAN performance [closed]

I am trying to evaluate the performance of the GAN model, I have trained. I found that there exist two major choices FID (Fréchet inception distance) and MMD (Maximum Mean Discrepancy) for comparing ...
Rajesh Nakka's user avatar
2 votes
1 answer
96 views

Evaluate upper bound prediction results using classic error calculation instead of PI metrics

I have the following data over time: that means data collected for a single variable like CPU usage in lowest, highest, and average mode over time every 5 mins (data granularity = 5mins) like the ...
Mario's user avatar
  • 441
1 vote
0 answers
29 views

Metric for inter-model agreement?

I have a classification task, with 3 classes. A model $M$ is a classifier that outputs the probability of the input being each of these classes. For example, for input $x$, I have $M(x)=[0.1, 0.2, 0.7]...
whoisit's user avatar
  • 101
3 votes
1 answer
87 views

What metric should I use for a Regression model with a gamma distributed target?

Background I'm building a regression model on insurance data to predict the losses associated with a policy. I'm running an Optuna optimisation function to help me with this, but I'm struggling with ...
Connor's user avatar
  • 635
3 votes
1 answer
31 views

Converting Adjusted R²

I just examined the $R^2_\text{adj}$ Formula on Wikipedia and found two ways to calculate the adjusted $R^2$. Firstly as $$R^2_\text{adj}=1-\frac{\frac{SS_\text{res}}{(n-p-1)}}{\frac{SS_\text{tot}}{(n-...
Linus's user avatar
  • 53
1 vote
1 answer
158 views

Symmetric AND Weight MAPE Calculation

I'm responsible to forecast a portfolio of consumer products on a monthly basis, and in calculating forecast accuracy, I'm lead to the MAPE (Mean Average Percent Error), which is useful, but has, ...
Mark J's user avatar
  • 11
1 vote
0 answers
16 views

When is PR curve more informative than ROC curve?

I am reading the paper A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks and the section 2 discusses the properties of AUROC vs AUPR. Some conclusions in the ...
longpollehn's user avatar
1 vote
0 answers
26 views

Evaluate two datasets for corresponding to given pairwise matching

Suppose I have two matched $N \times D$ datasets $X$ and $Y$. The samples are in matched order (with $n$th sample in $X$ being the same object as the $n$th sample in $Y$). The features are (or at ...
Betterthan Kwora's user avatar
2 votes
0 answers
41 views

Statistical Estimation of Machine Learning Metrics

Aloha Cross Validated, In the context of binary classifiers we have a number of metrics like accuracy, precision, recall, AUROC, F1, etc. To show robustness of a model we can gain a confidence ...
Zain Jabbar's user avatar
0 votes
0 answers
34 views

How to calculate IoU metric if there is no mask

For example: YOLOv8 segmentation doesn't output any mask for concrete class if no masks was detected, but there is a mask for this class in the ground truth masks. I should set metric to zero for such ...
Dmitry  Sokolov's user avatar
1 vote
1 answer
30 views

Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)

Consider these two approaches to evaluating a classifiers performance: Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
another_student's user avatar
0 votes
0 answers
19 views

Are there any research papers which show why Wasserstein distance is better than Jensen-Shannon/KL_div/Bhattacharya distance for specific use cases?

I am trying to find reliable research work which show why displacement based metrics such as Wasserstein distance is a better suited metric than Jensen-Shannon distance in specific use cases and for ...
user17420392's user avatar
0 votes
0 answers
27 views

How to measure the difference between two distributions of the same family?

Kullback-Leibler divergence seems to be a frequently used "metric" to measure the difference between probability distributions, regardless of their respective families. However, I would like ...
Value_Investor's user avatar
1 vote
1 answer
69 views

Finding a source for the definition of "clustering accuracy"

In papers about unsupervised clustering I see a lot of references to a metric "clustering accuracy" or "unsupervised clustering accuracy" (ACC) which is usually defined as ...
Cyo's user avatar
  • 11
2 votes
1 answer
74 views

Test or training data? R², predicted R² and adjusted R²

I would like to understand the difference between simple R², predicted R², and adjusted R². I have done several research and readings, but the difference is still not clear to me. I have even reached ...
guest's user avatar
  • 21
4 votes
2 answers
347 views

Connection between LASSO regression and Taxicab Geometry

The following is written on the Wikipedia entry of Taxicab Geometry: The [taxicab] geometry has been used in regression analysis since the 18th century, and is often referred to as LASSO. ...
Abced Decba's user avatar
7 votes
2 answers
296 views

Measure of difference or distinctiveness that's comparable across both discrete and continuous variables

I am a product manager for a data analytics startup, and on our platform our clients end up with large tables of data about their customers which has many attributes, which are of mixed datatype: some ...
Steve Estes's user avatar
0 votes
0 answers
648 views

How to calculate FID for a set with a small number of images?

I need to evaluate my generative model using FID (Fréchet inception distance). However, the dataset of real images that I have only contains 2719 examples. I've read that the authors of the metric ...
nietoperz21's user avatar
0 votes
0 answers
146 views

Sample size calculation for AB testing - Non binomial ratio metric

I'm currently working on a sample size calculation for an upcoming AB test related to our mobile app. Up until now, I've been dealing with binomial metrics, such as the conversion rate, which is ...
Discipulus's user avatar
0 votes
1 answer
21 views

Why do we always consider 2-variable affinity metrics?

Clustering algorithms often use some affinity metrics to cluster a dataset. Given some data points $x_1, x_2, \ldots, x_n$, it is common to compare $x_i$ and $x_j$ using a function $f(x_i, x_j)$ which ...
Hugo's user avatar
  • 1
2 votes
1 answer
102 views

How to choose the output vector size for metric learning? [closed]

In metric learning of e.g., MNIST images, a CNN projects a 28 x 28 image into a $d$-dimensional vector which gets passed to a metric learning loss function: minimize the Euclidean or cosine distance ...
Jan300385a's user avatar
1 vote
1 answer
34 views

How can I compare two methods of measuring variables for a set of objects?

I have a set of objects for which I measured the variables in two different ways. How can I determine whether these datasets overlap well in the feature space? I was thinking about calculating some ...
kate allerton's user avatar
0 votes
0 answers
26 views

Does this type of metric have a common name?

I tend to use metrics like the following: "80% of all parcels are delivered within the given 60 min time period" or "In 90% of all emergency calls, the rescue service is on the scene ...
ivegotaquestion's user avatar
1 vote
1 answer
252 views

Relations between the energy distance and MMD

I was wondering if there's any relation between the two metrics. Both measure the distance between distributions (or samples of them). And they seem quite similar. The energy distance can be ...
Maverick Meerkat's user avatar
2 votes
1 answer
290 views

Any good metric for measuring multi-annotator agreement on an imbalanced dataset?

Is there an agreement method that would be well-suited for a data annotation task where: the labels are discrete classes each datapoint belongs to exactly one class (multi-class classification) each ...
Marek's user avatar
  • 143
0 votes
0 answers
74 views

Which metric to compare two probability density?

I need to compare two distribution $p$ and $q$. But I don't have access to the distribution $p$, I want to approximate it by distribution $q$ that I construct iteratively by choosing design point. ...
YP BARRY's user avatar
0 votes
0 answers
88 views

Hausdorff and IoU are stopped changing while dice metric is decreasing

I am facing a problem in Hausdorff and IoU, where they stop learning when reaching a specific value! While the loss and dice metric keeps changing. surface_distance also has a problem since it is ...
AMAS AL's user avatar
  • 33
0 votes
0 answers
92 views

What metrics could I use to compare multiple 2-dimensional histograms/sinograms?

I have a set of two-dimensional shapes, each represented by one or more closed paths, which enclose one continuous area. My objective is to establish a measure of similarity between these shapes. I do ...
Dugnom's user avatar
  • 101
1 vote
0 answers
231 views

Generalizing Inverse Variance Weighting?

Consider a generalization of inverse-variance weighting, where we choose normalized weights $w_i = \dfrac{\sigma_i^{-p}}{W_p}$ for some $p\geq 1$, where $W_p \equiv \sum\sigma_i^{-p}$. Then $p=2$ ...
Erik's user avatar
  • 111
0 votes
0 answers
21 views

Does this metric exist for clustering data based off its similarity matrix?

I'm working with a dataset where all I have is the similarity matrix (with values 0 to 1, 0 being no similarity, 1 being identical). After I assign the labels, I loop through the matrix and, for each ...
user386237's user avatar
1 vote
0 answers
17 views

Is there a term used to refer to the total number of positive predictions?

I'm not sure how else to put it, but I often use the sklearn.metrics.classification_report function in order to measure the performance of various classification ...
Sean's user avatar
  • 4,077
1 vote
1 answer
61 views

Applying classification model when training and inference populations are different

I am looking for ways of estimating or mitigating the risk of applying a classification model (say logistic regression for simplicity) in a certain population (the inference set) that is known to be ...
jpsca1293's user avatar
0 votes
1 answer
162 views

Is there an error metric that decreases the weight when the target is near zero?

As precipitation prediction models can only predict positive values, they won't be able to undershoot small values by much. When it comes to overshooting, there is no boundary. High precipitation ...
schefflaa's user avatar
2 votes
1 answer
43 views

Composed cosine similarity

I have the following problem. I have 3 vectors $u,v,w$ of n dimensions. I'm able to find cosine similarities between $u$ and $v$, and between $v$ and $w$: $cosine(u,v)$ and $cosine(v,w)$. Can i use ...
Chg's user avatar
  • 21
0 votes
0 answers
196 views

Why weighted F1-measure, precision and recall are always very similar to accuracy in my problem?

I'm adopting accuracy and macro and weighted averages of precision, recall and f-measure for evaluating my model in a multiclass problem with an imbalanced dataset. However, I noticed that weighted f1,...
Zaratruta's user avatar
  • 1,018
0 votes
1 answer
42 views

Find most unique image from pair scores of a set of Images

Long story short, I have a set of vectors for each image after training on a model. I'd like to find the most unique image from the scores generated by ...
Aniketh Reddimi's user avatar
0 votes
0 answers
17 views

Flipping inputs in multilabel classification

I have framed a classification problem as follows: I have $N$ items, and wish to predict a set of relevant tags for each out of $M$ tags. An item can have anywhere from 0 to $M$ applicable tags. To ...
John's user avatar
  • 1
1 vote
1 answer
38 views

Statistical argument for this cluster measure

This quick clustering score discussion presents the following single cluster scoring functions: $$ c_i = 1-\sqrt{\frac{\sum_{j}^{N}\left(1-\phi(i,j) \right)^2}{N-1}} $$ and $$ C = 1-\sqrt{\frac{\sum_{...
jman's user avatar
  • 123
0 votes
0 answers
40 views

From pointwise convergence to uniform: metrics

Let $\mu_\theta$ be the limit of an empirical measure $\mu_{n, \theta}$. $\theta \in \Theta$ and $\Theta$ is a compact set. Morever, the maps $\theta \rightarrow \mu_{n, \theta}$ and $\theta \...
Eryna's user avatar
  • 319
0 votes
1 answer
30 views

In a regression model, what does RSS divided by predicted response signify?

I know that RSS explains the deviation between the model and actual data by measuring the square of the difference between them. I found this metric in one of the questions sir gave me. What is the ...
Bijay's user avatar
  • 103

1
2 3 4 5
9