Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

A metric is a function that outputs a distance between 2 elements of a set & meets certain strict criteria (some 'distance' functions are not metrics).

0
votes
0answers
12 views

Assess the dependence of LDA on the random seed

New to LDA, I would like to be sure that my output (in the first the step, the word-per-topic probabilities) depends on the input merely, and is (somewhat) stable whatever start value for the first ...
2
votes
0answers
14 views

Can you compute errors bars from Precision and Recall?

I am performing an object detection task for counting cars in an image. I have the confusion matrix (TP, FP, FP, TN) of the model. I guess TN is just zero in this case, as we aren't detecting where a ...
1
vote
0answers
15 views

How correctly overcome problem with an infinity MAPE? [duplicate]

Should I choose an other metric or is the way to handle this problem?
2
votes
0answers
17 views

How are mean results on benchmark obtained when training neural networks?

In most neural network papers, networks are trained on a known database where state-of-the-art performance is known ("benchmark"). Whatever metric is chosen to illustrate the network's performance, ...
0
votes
0answers
16 views

clustering: what metrics can be used to measure both intra-class variance and inter-class variance, and what are their complexity?

I was considering using the silhouette score, however its complexity is $O(mn^2)$ which can quickly become prohibitive with the size of the dataset.
0
votes
0answers
9 views

Lorenz curve and inequality between sports teams

I was wondering if you could develop a metric to study the inequality for a given stat such as how unequal points are distributed throughout a team, or how unequal rebounds are distributed throughout ...
2
votes
1answer
44 views

Validation set early stopping on custom metric

I am wondering whether it is ok to monitor validation set performance using a metric which is not optimized by the training algorithm, but which makes more sense in your domain. As a concrete example,...
1
vote
1answer
28 views

Can Negative Predictive Value and Positive Predictive Values be the same?

Is there any scenario where a negative predictive value and a positive predictive value would be the same? Specifically, when using a neural network for binary prediction. Can this be a sign of ...
3
votes
1answer
76 views

Does it make sense to use an Early Stopping Metric like “mae” instaed of “val_loss” for regression problems?

I am performing a regression on a Dataset and try to replace a mathematical Model with a Neural Network. To avoid overfitting I decided to use the Early Stopping Callback Function of Keras. So far I ...
0
votes
0answers
16 views

Sentiment analysis on IMDB database and choose of metric [duplicate]

I am currently working on an extract of the IMDB available on http://ai.stanford.edu/~amaas/data/sentiment/ . Since i try to predict the label (between 1 and 10) related with each review i face a ...
0
votes
0answers
2 views

Searching a Standart Method for Validation of Vehicle Dynamics Systems

i am searching a standart method for model validation of vehicle dynamics systems. I have to compare datas from a hil test bench and real road measurements. In the end, i have to make a conclusion if ...
1
vote
3answers
90 views

When is an AUC score misleadingly high?

I have an algorithm which gives an AUC (area under the receiver operating curve) of 0.94. I mean, this is amazing, but... probably too amazing, considering the difficulty of the task I am working on. ...
0
votes
0answers
17 views

Use probabilities given by classifier to compute a metric

I've got many images of body fishes and I would like to caracterize their textures. For that I take some fishes with prototypal pattern (large bars, medium bars, small bars), I annotate them with that ...
0
votes
0answers
78 views

Jaccard distance for sets of different sizes

Is the Jaccard distance still a metric if applied to sets of different sizes? Or does it require the input sets are of the same length?
0
votes
0answers
15 views

Can cross entropy be employed for measuring effective mapping of cross-modal data?

I am working on a new metric for cross-modal retrieval which can measure the effectiveness of mapping two modalities on a manifold. However, the usual approach is to employ a distance metric and not ...
0
votes
1answer
22 views

the score to hope for when evaluating model by MAE, MSE or RMSE

when doing evaluation and optimization of model by MAE, MSE or RMSE what should we look at and compare our score to, as a baseline or acceptable score for our model. should we look at the best ...
0
votes
0answers
126 views

Metrics in rpart decision trees

I am currently working with decision trees in R, I am using caret library. Source code of rpart can be found here: https://github.com/cran/rpart/blob/master/R/rpart.R I understand how decision trees ...
0
votes
0answers
18 views

Which is the best distance metric in an Indicator matrix

Is it okay to use the $\chi^2-distance$ when we have a indicator matrix? With Indicator matrix I mean the complete dijuntive table that is used in the Multiple Correspondence analysis. I mea n, if we ...
0
votes
0answers
51 views

Measure agreement among experts in multi-label classification task

I was wondering whether there is a metric that can be used in order to compute the agreement, and therefore something like an upper bound for classifiers, among expert-labelled data. Assume there is ...
0
votes
1answer
33 views

Why RMSE over MAE for matrix factorisation?

I have been trying to compare several matrix factorization algorithms and I've noticed that all the papers and libraries I've seen measure the Root Mean Square Error(RMSE) when intuitively I would ...
0
votes
0answers
16 views

Performance metrics to measure the validity of predicted text?

I have an RNN word based model. The goal of this model is to predict a certain number of words (5000 in this case) given a seed input (15 words). In my case, the words represent xml markups. For ...
2
votes
0answers
24 views

Equivalent Forms of Wasserstein Metric

Let $F, G$ be two cumulative distribution functions (c.d.f.) over real numbers, the Wasserstein metric is defined as $$d_{p}(F,G)=\inf_{U,V}||U-V||_{p},$$ where the infimum is taken over all joint ...
1
vote
1answer
9 views

Classification problem: custom minimization metric to shift the focus of the model?

Assume a binary classification problem, with $1$ denoted as a "bad" outcome, and $0$ as a "good" outcome. If it's relevant, in the sample there are significantly more bads than goods, and this is the ...
1
vote
1answer
19 views

Is (1 - Coherence) a metric, at a given frequency?

I'm performing some signal analysis and was using coherence (magnitude-squared coherence) to inference signals similarity. Now, I need to extend the framework by introducing a metric. I was wondering,...
0
votes
0answers
60 views

Best way to compare CNN outputs vectors

I was training a CNN (which contains only convolutional an pooling layers) to extract features of a given image. Output vector size is ...
0
votes
1answer
43 views

R train function input multiple metrics [closed]

I want to get multiple metrics results like the code below but it didnt work because of metric = list("ROC","F1","Accuracy","Kappa"). It works perfect for ...
0
votes
0answers
45 views

Difference between the Wasserstein metric, mallows metric and Earth mover's distance

I'm really confused, is there a difference between the Wasserstein metric, mallows metric and Earth mover's distance? If yes What is it? Thank you
0
votes
1answer
71 views

When does the Wasserstein metric attain inequality WLOG?

I’m reading a classic paper [1] that describes a version of the Wasserstein metric (aka Mallows metric), defined as follows. Let $F$ and $G$ be probabilities in $\mathbb{R} ^B$, and let $U \sim F$ and ...
0
votes
0answers
15 views

Way to put variables on same scale across data sets

I have calculated four scores (say, A1/B1/C1/D1), where each score is made up granular variables in data set 1. I have calculated the same four scores (say, A2/B2/C2/D2) in data set 2. A1 and A2 are ...
2
votes
1answer
41 views

Comparison of Bayesian Neural Network with Multilayer Perceptron

I have a machine learning project with not so much data, so I have the following reasons to use Bayesian neural network (not Bayesian network / directed graphical models) for my work: There are ...
2
votes
1answer
173 views

Hyperparameter tuning in multiclass classification problem: scoring metric relevant?

I'm working with an imbalanced multi class dataset. I try to tune the parameters of a DecisionTreeClassifier, ...
0
votes
1answer
28 views

Correlation metric for 0-1 vector and real values vector [duplicate]

I am looking for a right correlation metric to measure correlation between two vectors. The dimensionality of the vectors is around 100. The first vector has real values as the elements and the second ...
0
votes
0answers
34 views

Need a very specific metric for distance between two vectors.

I have a very unusual requirement wrt a metric to measure distance between two vectors. I'm going to illustrate my problem with an example. Each row is a unique vector: the original, vec a & b....
1
vote
1answer
37 views

Choropleth visualisation to show variations from correlation

I'm trying to think of a good way to visualise a dataset which will consist of: a region identifier (for which I have geographies) a socioeconomic index expressed as a decile (this region is in the ...
1
vote
0answers
299 views

Proper way to use NDCG@k score for recommendations

Currently I am building a recommender system and using ranking metrics to verify its performance. I am using the NDCG@k score. Today I was experimenting and I realized that I might be calculating the ...
1
vote
0answers
131 views

Is it always better to use F1-Score than Accuracy as performance metric?

During reading papers about Machine Learning I always find researchers using accuracy as their sole performance metric. However, a high accuracy alone proves nothing when the amount of false positives ...
3
votes
1answer
54 views

The trace term in 2 Wassersteins metric for Gaussians

I was looking at the formula for 2 Wassersteins distance for Gaussian distribution on Wikipedia. https://en.wikipedia.org/wiki/Wasserstein_metric#Normal_distributions It satisfies all properties of a ...
3
votes
0answers
77 views

calculating orientation anisotropy of a set of 3D vectors

I have a set of unit-length 3D vectors (represented as cartesian 3-tuples) with 180$^\circ$ equivalence (rotated through any plane passing through the origin i.e. $v=-v$), all situated at the origin (...
0
votes
0answers
26 views

estimate score on test set when ground truth is available only for a part of it

I would like to evaluate the performance of my machine learning model on a test set, but I only have access to the ground truth for a subset of the test set, say 60% of it. On the remaining 40%, the ...
0
votes
0answers
14 views

metric score when ranking information is only available at test time

I am training a machine learning model where each training sample consists of a set of "competitors" and the aim is to predict the winner (there might be more than one winner. I don't care about ...
0
votes
1answer
80 views

Which is the correct F-beta-measure scoring formula?

I have found differing formulas to calculate the $Fbeta$ score, which changes the weight (influence) of $PPV$ ($Precision$) or $TPR$ ($Recall$) from the default of equality (i.e. $0.5$). For example: ...
0
votes
0answers
16 views

Why isn't the performance metric Geometric-measure/G-measure/G1 score used more often, instead of F1?

From wikipedia: While the F-measure is the harmonic mean of Recall and Precision, The G-measure is the geometric mean. G1 = Sqrt ( PPV x TPR ) F1 = 2 / ( ( PPV ...
0
votes
0answers
20 views

Is there a metric to measure “degree of deviation” between two strings?

I'm trying to figure out a metric that can estimate the "degree of deviation" between two strings. For example, if I have two strings such as "abcdfhg" and "aaeefhf", I want to calculate how much the ...
0
votes
0answers
79 views

Evaluating a binary classification time series model

Question: What metric should I use to compare time series models that are used for binary classification when the classes are highly imbalanced? Is there a way to get an average score that makes sense ...
3
votes
0answers
47 views

Is it dangerous to take the mean of ratio measurements?

I'm going through a Computer Architecture course and in one of the lectures they mention that when measuring the difference in runtime performance between two systems, using for example multiple ...
1
vote
1answer
79 views

Map distance between two time series to a probability

We have several time series, which are basiclly chunks of numeric values. We use Dynamic Time Warping to calculate the distance between these time series. This is working well and gives us some ...
3
votes
0answers
97 views

Prove the existence of a fixed point of a certain mapping of distributions

Let $\tilde{X}_0$ be some random variable on $\mathbb{R}^n$, with a strictly positive p.d.f.. Define: $$X_0:=(\operatorname{var}{\tilde{X}_0})^{-\frac{1}{2}}(\tilde{X}_0-\mathbb{E}\tilde{X}_0),$$ ...
2
votes
0answers
318 views

Clustering in high dimensions: distance metrics, binary vs continuous, statistical tests for number of clusters / noise points [closed]

I’ve got several thousand observations in approximately 300-dimensional space, in a relatively sparse matrix (typically 30 non-zero dimensions per observation). I'm using a clustering algorithm (so ...
0
votes
0answers
27 views

How should I describe a correct answer is within Top-K

I have a classification task which predicts class id with a probability. I want to show the accuracy which the correct label is included with top-K in my paper. I want to know the name of this metrics....
1
vote
1answer
2k views

XGBClassifier and XGBRegressor

I am a newbie to Xgboost and I would like to use it for regression, in particular, car prices prediction. I started following a tutorial on XGboost which uses XGBClassifier and objective= 'binary:...