Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [metric]

A metric is a function that outputs a distance between 2 elements of a set & meets certain strict criteria (some 'distance' functions are not metrics).

0
votes
0answers
9 views

Replacement for angular distance metric

I am looking for a distance metric that could be used instead of cosine/angular distance for high dimensional data. Metric that is limited the same way as cosine/angular distance is would be great. ...
0
votes
0answers
17 views

Consistency metric explanation

I am trying to understand a bit more about the consistency metric (to understand how consistency-based subset evaluation works). I find on this paper the following equation : $$\text{Consistency}_s =...
1
vote
0answers
13 views

Suitable performance metric for an unbalanced multi-class classification problem?

I have an unbalanced multi-class classification problem with the following class distributions: Class 0: 17.1% Class 1: 63.2% Class 2: 19.7% I am using scikit-...
0
votes
0answers
10 views

Mahalanobis: Covariance or correlation?

Is the Mahalanobis distance using correlations or covariances (among two vectors) to determine the similarity? I know that both are quite the same and differ only by a division of a standard deviation ...
0
votes
0answers
8 views

Which performance metric to use for stratified data? [duplicate]

I'm trying to classify a data into 3 classes (supervised), one of which is heavily underrepresented in the data set. In order to combat this imbalance, I decided to stratify the data. Now I want to ...
0
votes
0answers
16 views

Is recall a relatively meaningless metric in a balanced dataset?

Just looking for a sanity check here. Leaving aside precision, is talking about the recall of a binary classification algorithm sensible where 50% of the cases presented to it are positive and 50% ...
0
votes
0answers
10 views

Machine learning: eval_function for multiclass classifcation

I have checked What are the measure for accuracy of multilabel data? but I think the discussion somehow confused between loss and evaluation function. Say I have a multi-class classification problem ...
1
vote
2answers
18 views

Metrics for unbalanced classes [duplicate]

I have been looking for good metrics on this data set I am working, however it is highly unbalanced. It has a total of 8 categorical classes, one of them is responsible for ~40% of data, another for ...
2
votes
2answers
30 views

Localized distance function on sequential binary data

I am trying to find a good distance function for sequential data that is all binary. For now, I am using Edit distance however I have some more domain-specific knowledge that I would like to ...
0
votes
0answers
7 views

Quality metric of sampled time series data

I have a time series that has too many points. I sample one in every 100 points, in order to reduce the amount of data I need to transmit from my measurement device. What accuracy metric can I use ...
1
vote
1answer
46 views

Imbalanced Test Data

I have an imbalanced (1:5) training and test set with only two classes and have oversampled the training set with SMOTE so that the class ratio is 1:1. The ML model gives values over 0.7 for accuracy, ...
0
votes
0answers
21 views

Measure for evaluating a generated ranking of items, when best k items are known

I am evaluating different kinds of feature selection algorithms. Some return a ranking of variables based on their usefulness (to predict a target variable, etc.). Lets say there are n variables in ...
0
votes
0answers
11 views

Understanding the Gini/AUC metric as out-of-development performance metric

Assume we develop a model for a binary classification task that reaches a certain Gini/AUROC estimate on the validation ( or training ) sample, among others. This is an overall good metric, often used ...
2
votes
2answers
207 views

Loss function and evaluation metric

When building a learning algorithm we are looking to maximize a given evaluation metric (say accuracy), but the algorithm will try to optimize a different loss function during learning (say MSE/...
0
votes
0answers
25 views

Best performance metric for highly imbalanced dataset f1 score vs kappa vs AUROC

I have highly imbalanced data (like fraud detection). I usually use f1 score to evaluate model performance. But I also saw people recommend AUROC and cohen's kappa. I'm seeking expert opinion on what ...
0
votes
0answers
14 views

Correct way of calculating cohort-based daily retention metric

Background: I need to calculate retention of users observed in a system. I will use this metric to estimate the Bayesian probability of said system's retention levels in the future. This question is ...
0
votes
0answers
5 views

Does zero-padding truth and estimate deteriorate the NRMSE/NRMSD?

I feel like I am missing an important point, but maybe the NMRSE (or NMRSD) is just counter-intuitive. This is the equation that I use: $\textrm{NMRSE} = \frac{\sqrt{\sum_i \left( \textrm{est}_i - \...
1
vote
1answer
115 views

different trends in loss and AUC ROC metric

I am training a deep neural network for a binary classification I am using binary_crossentropy as loss and area under the roc curve as performance metric as ...
0
votes
0answers
16 views

is there a measure for the roughness of a contour plot

There has to be a measure for the difference between "instantaneous" change of "energy" along a line in a space compared to averaged changed of energy along a line. I could take a smooth surface in ...
1
vote
1answer
35 views

Can two imperfect rankings be combined to produce a better ranking?

So I've got a dataset that can be ranked in two noisy ways $R_1$ and $R_2$. Let's call $R_1$ and $R_2$ functions from the dataset $x \in R^N$ to a real number between 0 and 1. Since they are rankings ...
0
votes
0answers
26 views

ranking metric for a specific use case

I am trying to find the best "ranking metric" for my problem but could not find any existing suitable metric, can someone help. I'm interested in information retrieval. For item there is a "...
0
votes
1answer
96 views

Assess the dependence of LDA on the random seed

New to latent Dirichlet Allocation (LDA), I would like to be sure that my output (in the first the step, the word-per-topic probabilities) depends on the input merely, and is (somewhat) stable ...
2
votes
0answers
28 views

Can you compute errors bars from Precision and Recall?

I am performing an object detection task for counting cars in an image. I have the confusion matrix (TP, FP, FP, TN) of the model. I guess TN is just zero in this case, as we aren't detecting where a ...
1
vote
0answers
17 views

How correctly overcome problem with an infinity MAPE? [duplicate]

Should I choose an other metric or is the way to handle this problem?
2
votes
0answers
25 views

How are mean results on benchmark obtained when training neural networks?

In most neural network papers, networks are trained on a known database where state-of-the-art performance is known ("benchmark"). Whatever metric is chosen to illustrate the network's performance, ...
0
votes
0answers
28 views

clustering: what metrics can be used to measure both intra-class variance and inter-class variance, and what are their complexity?

I was considering using the silhouette score, however its complexity is $O(mn^2)$ which can quickly become prohibitive with the size of the dataset.
0
votes
0answers
13 views

Lorenz curve and inequality between sports teams

I was wondering if you could develop a metric to study the inequality for a given stat such as how unequal points are distributed throughout a team, or how unequal rebounds are distributed throughout ...
2
votes
1answer
162 views

Validation set early stopping on custom metric

I am wondering whether it is ok to monitor validation set performance using a metric which is not optimized by the training algorithm, but which makes more sense in your domain. As a concrete example,...
1
vote
1answer
29 views

Can Negative Predictive Value and Positive Predictive Values be the same?

Is there any scenario where a negative predictive value and a positive predictive value would be the same? Specifically, when using a neural network for binary prediction. Can this be a sign of ...
3
votes
1answer
113 views

Does it make sense to use an Early Stopping Metric like “mae” instaed of “val_loss” for regression problems?

I am performing a regression on a Dataset and try to replace a mathematical Model with a Neural Network. To avoid overfitting I decided to use the Early Stopping Callback Function of Keras. So far I ...
0
votes
0answers
18 views

Sentiment analysis on IMDB database and choose of metric [duplicate]

I am currently working on an extract of the IMDB available on http://ai.stanford.edu/~amaas/data/sentiment/ . Since i try to predict the label (between 1 and 10) related with each review i face a ...
1
vote
3answers
124 views

When is an AUC score misleadingly high?

I have an algorithm which gives an AUC (area under the receiver operating curve) of 0.94. I mean, this is amazing, but... probably too amazing, considering the difficulty of the task I am working on. ...
0
votes
0answers
136 views

Jaccard distance for sets of different sizes

Is the Jaccard distance still a metric if applied to sets of different sizes? Or does it require the input sets are of the same length?
0
votes
0answers
20 views

Can cross entropy be employed for measuring effective mapping of cross-modal data?

I am working on a new metric for cross-modal retrieval which can measure the effectiveness of mapping two modalities on a manifold. However, the usual approach is to employ a distance metric and not ...
0
votes
1answer
32 views

the score to hope for when evaluating model by MAE, MSE or RMSE

when doing evaluation and optimization of model by MAE, MSE or RMSE what should we look at and compare our score to, as a baseline or acceptable score for our model. should we look at the best ...
0
votes
0answers
277 views

Metrics in rpart decision trees

I am currently working with decision trees in R, I am using caret library. Source code of rpart can be found here: https://github.com/cran/rpart/blob/master/R/rpart.R I understand how decision trees ...
0
votes
0answers
19 views

Which is the best distance metric in an Indicator matrix

Is it okay to use the $\chi^2-distance$ when we have a indicator matrix? With Indicator matrix I mean the complete dijuntive table that is used in the Multiple Correspondence analysis. I mea n, if we ...
0
votes
0answers
116 views

Measure agreement among experts in multi-label classification task

I was wondering whether there is a metric that can be used in order to compute the agreement, and therefore something like an upper bound for classifiers, among expert-labelled data. Assume there is ...
0
votes
1answer
92 views

Why RMSE over MAE for matrix factorisation?

I have been trying to compare several matrix factorization algorithms and I've noticed that all the papers and libraries I've seen measure the Root Mean Square Error(RMSE) when intuitively I would ...
0
votes
0answers
17 views

Performance metrics to measure the validity of predicted text?

I have an RNN word based model. The goal of this model is to predict a certain number of words (5000 in this case) given a seed input (15 words). In my case, the words represent xml markups. For ...
2
votes
0answers
35 views

Equivalent Forms of Wasserstein Metric

Let $F, G$ be two cumulative distribution functions (c.d.f.) over real numbers, the Wasserstein metric is defined as $$d_{p}(F,G)=\inf_{U,V}||U-V||_{p},$$ where the infimum is taken over all joint ...
2
votes
1answer
13 views

Classification problem: custom minimization metric to shift the focus of the model?

Assume a binary classification problem, with $1$ denoted as a "bad" outcome, and $0$ as a "good" outcome. If it's relevant, in the sample there are significantly more bads than goods, and this is the ...
1
vote
1answer
25 views

Is (1 - Coherence) a metric, at a given frequency?

I'm performing some signal analysis and was using coherence (magnitude-squared coherence) to inference signals similarity. Now, I need to extend the framework by introducing a metric. I was wondering,...
0
votes
0answers
155 views

Best way to compare CNN outputs vectors

I was training a CNN (which contains only convolutional an pooling layers) to extract features of a given image. Output vector size is ...
0
votes
1answer
175 views

R train function input multiple metrics [closed]

I want to get multiple metrics results like the code below but it didnt work because of metric = list("ROC","F1","Accuracy","Kappa"). It works perfect for ...
0
votes
0answers
84 views

Difference between the Wasserstein metric, mallows metric and Earth mover's distance

I'm really confused, is there a difference between the Wasserstein metric, mallows metric and Earth mover's distance? If yes What is it? Thank you
0
votes
1answer
91 views

When does the Wasserstein metric attain inequality WLOG?

I’m reading a classic paper [1] that describes a version of the Wasserstein metric (aka Mallows metric), defined as follows. Let $F$ and $G$ be probabilities in $\mathbb{R} ^B$, and let $U \sim F$ and ...
0
votes
0answers
16 views

Way to put variables on same scale across data sets

I have calculated four scores (say, A1/B1/C1/D1), where each score is made up granular variables in data set 1. I have calculated the same four scores (say, A2/B2/C2/D2) in data set 2. A1 and A2 are ...
2
votes
1answer
98 views

Comparison of Bayesian Neural Network with Multilayer Perceptron

I have a machine learning project with not so much data, so I have the following reasons to use Bayesian neural network (not Bayesian network / directed graphical models) for my work: There are ...
2
votes
1answer
250 views

Hyperparameter tuning in multiclass classification problem: scoring metric relevant?

I'm working with an imbalanced multi class dataset. I try to tune the parameters of a DecisionTreeClassifier, ...