Questions tagged [metric]

A metric is a function that outputs a distance between 2 elements of a set & meets certain strict criteria (some 'distance' functions are not metrics).

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

What are some good relative regression metrics?

Which relative regression metrics exist? What are their strengths and weaknesses? In what case do you use each? --> Bonus point if they are already/easily implemented in Scikit-Learn. I have a ...
user avatar
2 votes
2 answers
50 views

XGBRegressor score (R2) vs. eval_metric (RMSE)

According to the API Reference, XGBRegressor().score() returns R2. However, according to the XGBoost Paramters page, the default ...
user avatar
  • 21
2 votes
2 answers
49 views

Is there a metric to compare machine learning models which takes into account how many trainable parameters they have?

Assume we have two models for the same task. One with 1K parameters and one with 1M and they achieve accuracy as follows Model Params Accuracy small 1 000 0.8 big 1 000 000 0.9 I would like to ...
user avatar
0 votes
0 answers
54 views

Wasserstein distance between multivariate lognormal distributions

Wikipedia gives the following formula for normal distributions: What changes, if any, do I need to make to handle multivariate lognormal distributions instead?
user avatar
  • 23
1 vote
0 answers
17 views

Evaluate two lossy compression algorithms

I am trying to evaluate several methods to compress some 2D data points. The algorithm itself is not relevant, but from the output, I can compute the MSE and the number of points (which can be used to ...
user avatar
0 votes
0 answers
15 views

Using Inception and FID scores in training?

Is it possible to use the Inception and FID scores in the training of a deep image generation model, i.e. to maximize the scores in a loss function, albeit this is "cheating"? If so, has ...
user avatar
  • 1
2 votes
1 answer
128 views

How to evaluate complementary datasets for ML models?

Evaluating ML models is a fundamental task and subfield of the Machine Learning practice. On the other hand, I was not able to find any existing materials, guides, protocols, papers on how to proceed ...
user avatar
0 votes
0 answers
31 views

Can the chi-square statistic in Kruskal-Wallis test be compared to determine the most appropriate to distinguish the groups?

I have a dataframe in R which format is similar as follows: v1 v2 v3 group 1 3.5 100 a 3 5 200 a 10 5.5 150 b 8 7.5 210 b 4 4.5 300 c 9 2.5 200 c ... My ...
user avatar
  • 1
0 votes
1 answer
26 views

What metrics work well in unbalanced assemblies?

I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
user avatar
0 votes
0 answers
67 views

Metric to quantify randomness

I have an $L \times L$ matrix representing a $2D$ region. Each entry of this matrix is a real number lying in the closed interval [0,1]. I want to quantify how different is this from a similar $L \...
user avatar
  • 1
1 vote
0 answers
17 views

Metrics for imbalanced multi-class classification [duplicate]

I am looking for informations about metrics for classification with 3 unbalanced classes. I have following numbers of samples in every class: 1 As you can see two classes are quite balanced and one is ...
user avatar
  • 31
1 vote
1 answer
90 views

Why does contrastive loss distinguish positive from negative samples?

Trying to learn Siamese networks for ranking tasks from here, I find it hard to understand why the contrastive loss is not symmetric for positive and for negative examples. The contrastive loss $L(A, ...
user avatar
  • 373
0 votes
0 answers
12 views

metrics used in search verification, search engines

It's a rather open ended question I think. I'm working on a search engine, and I'd like to quantify my algorithm's performance. I'm aware of search ranking algorithms like pagerank, etc. but the ...
user avatar
  • 241
0 votes
0 answers
3 views

What metric is best fitted for comparing encodings?

I am trying to compare two distributions, that each correspond to different numerical encodings, e.g. compare fp32 encodings to various other encodings on a same set of values. However I do not know ...
user avatar
1 vote
1 answer
36 views

Is there a good reason for using AUROC on imbalanced dataset?

So I just learned about AUROC. When I read this thread, it seems like AUROC is not a great metric for imbalanced dataset. One answer even says it shouldn't be used to compare models. However, I am ...
user avatar
  • 135
0 votes
0 answers
27 views

Accuracy for unfairness detection

I am dealing with unfairness and I am trying to find out some metrics to detect unfairness presence in my dataset. I am starting from the very bases: accuracy. My main question is Do you think I can ...
user avatar
0 votes
0 answers
9 views

Equal Error Rate in Hidden Markov Models for Speaker Recognition

I was asked to report the Equal Error Rate (EER) for my speaker recognition proposal. I trained one HMM for each speaker. To evaluate, I introduce the input of an speaker in both models and classify ...
user avatar
  • 83
2 votes
1 answer
103 views

If two distributions have the same moments, how different can they be?

Let us suppose we have two distribution functions $F$ and $G$ with shared domain and also shared moments but not necessarily shared moment-generating functions. I have seen from "Whether ...
user avatar
  • 3,546
0 votes
0 answers
27 views

Why do we sort confidence scores when calculating MAP in Object Detection?

This question is specific to object detection's metric Mean Average Precision. I sieved through multiple articles like the one by Jonathon and here by Kaggler Tito. One part I cannot grasp immediately ...
user avatar
  • 533
0 votes
0 answers
36 views

Why is KL divergence used as a measure of closeness in variational inference?

I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).
user avatar
  • 223
1 vote
1 answer
36 views

Comparing AUROC vs F1 score

I would like to ask a question on how to interpret the results of two different models, based on AUROC and F1 metrics. As all of you know, AUROC calculates the area under the ROC curve, and the F1 ...
user avatar
  • 583
0 votes
0 answers
15 views

Seeking A Scale-Independent Alternative To Q2 For Model Selection When The Response Varies Over Multiple Orders of Magnitude

I am using constrained polynomial regression to predict y = f(x). I have prior knowledge about the relationship that allows me to add constraints to the optimization problem for the first and second ...
user avatar
0 votes
1 answer
38 views

Metric to measure how "standard Gaussian" a set of samples is?

Assume that I have a set of $N\in\mathbb{R}^{D}$ samples from some otherwise unknown multivariate distribution $p$. I seek a metric which might tell me how "close" $p$ is to a standard ...
user avatar
  • 421
0 votes
0 answers
16 views

Assess results of non-linear continous model

I have a set of y_true labels and a set of y_pred labels. The prediction model is actually a deep learning model in combination with a rule-based information extractor. It tries to find the ...
user avatar
  • 1
0 votes
0 answers
23 views

What metric can be considered to find if causal impact has determined the best combination of synthetic controls

I'm new to Causal impact. I read the paper and video by Kay which has a detailed description of the package. Can someone suggest any metric which can describe the accuracy of the synthetic control ...
user avatar
0 votes
1 answer
36 views

How do I obtain a percentage accuracy for an LSTM

I've got an LSTM trained for time series forecasting and I've seen people online report their LSTMs accuracy in %s such as 85% accurate etc, how do I obtain a metric like this? so far I was just using ...
user avatar
0 votes
2 answers
118 views

Mediation analysis with dichotomous, categorial and metric data

I try to run a Mediation analyses. Unfortunately PROCESS by Hayes doesn't allow dichotomous variables. How can I solve the Problem? How can I calculate for direct and indirect effects? My variables ...
user avatar
3 votes
2 answers
216 views

Alternative to mean absolute percentage error (MAPE)

MAPE metric has problems when the actual value to be predicted is very small. In the extreme when the actual value is 0 then MAPE will be infinity (if the prediction is not exactly 0). What about this ...
user avatar
  • 923
-1 votes
1 answer
119 views

How to use siamese network in binary classification - inference mode

Siamese network consists of two identical networks. Networks share the same weights. The general workflow is as follows (taken from here): Suppose that I have 10 images of apples, 10 images of ...
user avatar
  • 199
2 votes
2 answers
112 views

How many samples are needed to estimate quantiles for an unknown distribution?

I'm trying to evaluate performance of a metric learning model. The model that takes labelled image inputs and maps them to vectors on an N-dimensional unit sphere. The goal of the model is to map ...
user avatar
1 vote
1 answer
18 views

What is good consolidated/normalized metric with funnel page views data?

I am currently working on a best way to represent product interest score based on page views. Working on an ecommerce use case having homepage, search page, and product page interaction data available....
user avatar
  • 11
1 vote
0 answers
35 views

Distance defined by second moment, akin to Mahalanobis distance?

In ordinary linear regression ($c=0$) and ridge regression ($c > 0$), for design matrix $X$ with dimensions $N$ observations by $D$ dimensions, the $N \times N$ hat matrix is given by: $$H = X (X^T ...
user avatar
2 votes
1 answer
123 views

Is there an official procedure to compute mIoU (mean intersection over union)?

Although it sounds silly, I'm not finding an official source to compute mean intersection over union (mIoU). I'm realizing a semantic segmentation task, and I want to compute the mIoU over a dataset. ...
user avatar
0 votes
0 answers
32 views

A distance function between music playlists

I am looking for a way to measure the dissimilarity/distance $d$ between a set of music playlists $\{P_i\}$ with possibly different number of songs. We may assume that a playlist contains a specific ...
user avatar
0 votes
0 answers
9 views

Best Error Metric for Quantity Spread Estimation Accuracy

What is the best error metric for the accuracy of predicting how a fixed quantity is spread over several buckets? E.g. say there were 10 units that needed to be spread across three categories: ...
user avatar
  • 171
3 votes
1 answer
54 views

Scale Invariant Statistical Distances

Problem Suppose we have empirical distributions to two $n$-dimensional random variables $X = (X_1, X_2, ..., X_n)$ and $Y = (Y_1, Y_2, ..., Y_n)$. The goal is to find $k < n$ components, such that ...
user avatar
  • 133
0 votes
2 answers
54 views

What is a good metric to evaluate ranking without retrieval?

The problem is predicting the ranking of a list using some features e.g. rank_metric([A, B, C, D, E], [A, B, C, D, E]) should be 1 ...
user avatar
3 votes
2 answers
62 views

What is the name for this metric?

I have been using the metric shown below for a while, in a power forecasting context. We call it WAPE (Weighted Average Percentege Error, but I have never seen any ...
user avatar
  • 31
1 vote
1 answer
78 views

How does the Bhattacharyya distance doesn't satisfy triangle inequality?

Googling doesn't seem to show many informative results. I don't know if the concept is too trivial that I should know immediately or it's an old topic. It's either article / blogs repeating the wiki ...
user avatar
0 votes
0 answers
57 views

Why isn't ROUGE-N normalized by the number of N-grams in the reference summary?

Note: I'll focus on $ROUGE-1$, but the same holds for $ROUGE-N$. For a machine-produced summary $M$ and a bunch of reference summaries $RefSummaries$, I believe $ROUGE-1$ can be calculated in the ...
user avatar
0 votes
1 answer
16 views

Usage of tracking signal as a metric when number of forecasts is low

Suppose Tracking Signal (TS) is used as a metric to evaluate the quality of a forecast. Let a be the ground truth value and f be ...
user avatar
0 votes
0 answers
23 views

Metric for comparing supervised and unsupervised model

I'm searching on how to compare (validate) a supervised learning model to an unsupervised one. Let's say I have a supervised model for fault diagnosis which can tell me how accurate it is to predict ...
user avatar
  • 1
2 votes
1 answer
80 views

Is negative log likelihood calculated in log space or exponential space?

I have a question about calculating negative log likelihood in a machine learning model over a dataset which seems simple but I cannot find a solid answer/explanation online. Is the NLL calculated as ...
user avatar
  • 629
0 votes
0 answers
5 views

Intrinsic metrics to evaluate pretrained language models?

I learned that there are intrinsic and extrinsic evaluation methods for vector models. Although the most important evaluation is the extrinsic the intrinsic metrics are also useful. There are three ...
user avatar
  • 5,286
0 votes
0 answers
6 views

Best way to take into account the number of labeled data to be used for semi-supervised classification task

I am having two datasets with features drawn from different distributions. Both datasets contain equal classes for the same task. The thing i am facing is called domain adaptation - so i train a ...
user avatar
  • 1
1 vote
0 answers
48 views

Why is AUC so often use to compare performance of different models in churn prediction task?

I have to build model to predict churn and when reading related work on the internet I have realized that in most of the cases the AUC is used as a metric to compare different models. That's ...
user avatar
  • 75
0 votes
0 answers
67 views

Is the expected distance of two random variables a Metric?

This is purely for more understanding and not an assignment. I understand the definition of a metric https://en.wikipedia.org/wiki/Metric_(mathematics). I also understand that the following is a ...
user avatar
  • 31
0 votes
1 answer
61 views

In general, what are precision, recall, F1 that are reported in papers?

I used classification_report in sklearn library And, the picture below shows evaluation on my model (anomaly detector) In general, what are precision, recall, F1 ...
user avatar
1 vote
0 answers
21 views

Which performance metric can I use for (multi) aspect based sentiment analysis?

I'm working on creating a model that extracts and evaluates sentiments of aspects in the text. My problem is that I'm unsure how to evaluate my results. Currently, I'm looking at the sentiment and if ...
user avatar
  • 11
0 votes
0 answers
12 views

How to verify whether a metric is of negative type or not?

A metric $d(\cdot,\cdot)$ of a space $S$ is said to be of negative type, if for $\forall n \geq 2, z_{1}, \ldots, z_{n} \in S$, and $\alpha_{1}, \ldots, \alpha_{n} \in \mathbb{R}$ with $\sum\limits_{i=...
user avatar
  • 125

1
2 3 4 5
8