Questions tagged [metric]

A metric is a function that outputs a distance between 2 elements of a set & meets certain strict criteria (some 'distance' functions are not metrics).

Filter by
Sorted by
Tagged with
1
vote
0answers
20 views

Similarity measure in classification: when do you need it to be a metric?

I am currently considering using dynamic time warping (DTW) for outlier detection in time series data. I am fairly new to the technique and have been reading: That I shoud be cautious using DTW since ...
0
votes
0answers
14 views

How to compute Net Reclassification Improvement or Alternatives?

I am working on a binary classification problem with ~5k records and class proportion of 33:67. I have 60 features/variables in my dataset and finally I have come to about 10 variables based on ...
0
votes
0answers
8 views

In the multi-class case micro-precision=micro-recall=micro-F1=accuracy

The statement is from https://towardsdatascience.com/multi-class-metrics-made-simple-part-ii-the-f1-score-ebe8b2c2ca1 my question is whether in some papers this statement was mentioned and also ...
0
votes
0answers
40 views

validation of survival analysis model

I am currently looking for evaluating/validating a survival analysis model on quite highly right censored data set. The thing is that i have many individuals in the data set. I wanted to use c-index ...
6
votes
4answers
1k views

What is a metric can I use to calculate the distance between labels?

Let's say we have a set of labels of the same length, and we need to find the distance between them. In the case of binary labels, one can use the Hamming distance. For example, if $l_1 = 01101$ and $...
0
votes
0answers
18 views

Internal clustering criteria as a measure to know if feature transformation is useful

Does it make sense to use internal clustering criteria, such as Calinski-Harabasz, Davies-Bouldin and Silhouette, to draw conclusions about whether feature scaling makes sense with the existing data? ...
0
votes
0answers
14 views

Confidence Intervals for the Classification Accuracy

I am developing a classification system and after some iterations I settled on a Random Forest algorithm as the final predictor. I would like to have the confidence intervals for the estimated model's ...
0
votes
0answers
25 views

Is there a name for this metric?

I'm testing different methods for carrying out variant calling with HIV sequencing data and want to compare the performance of each method. I have true and false positive counts for each method but ...
1
vote
0answers
19 views

Ranking metric that takes into account length of result list?

I would like to evaluate a problem where the user select an option from a list of variable length. The task is to provide a ranked list so that the item lower in rank is the most relevant. If I use a ...
0
votes
1answer
61 views

In k-means or kNN, we use euclidean distance to calculate the distance between nearest neighbours. Why not manhattan distance?

In k-means or kNN, we use euclidean distance to calculate the distance between nearest neighbours. Why not manhattan distance ?
1
vote
0answers
14 views

Determine outliers for robust Mahalanobis distance

I want to apply a robust mahal distance and found an implementation in scikit: https://scikit-learn.org/stable/auto_examples/covariance/plot_mahalanobis_distances.html but there is the number of ...
2
votes
1answer
134 views

Fleiss kappa vs Cohen kappa

Can somebody explain in-detailed differences between Fleiss kappa and Cohen kappa? And how the metric works under the hood? When would one use Fleiss kappa over Cohen kappa? What are the ...
1
vote
3answers
35 views

Binary metric for an unknown, but highly imbalanced, data ratio?

I'm looking for a good metric to compare binary classification methods for a task where The data is highly imbalanced. The approximate data imbalance is unknown. There are certainly more than 100 ...
0
votes
0answers
49 views

What is the difference of “normal” F1 and macro average F1 score with binary classification

Please note that I always talk about binary classification here. I do not speak about multi class classification. In case of unbalanced binary datasets it is a good practice to use F1 score. While ...
2
votes
1answer
23 views

Which evaluation metrics are mutually redundant?

Suppose we are given a confusion matrix for a binary classification: tp, fp fn, tn Now, there are lots of evaluation metrics: POD (probability of detection, aka hit rate, ...
0
votes
0answers
15 views

measure of the difference between variation of two time series of probability distributions

I'm looking at a series of particle density probabilities of proteins floating on a cell, these particles move around and also blink, so these density probabilities differ a bit from one time to ...
1
vote
0answers
18 views

Sum R-Squared over uncorrelated features

I am currently developing an automatic approach to eliminate noisy samples from a data set. I clustered all the samples, and then I am iterating over all the clusters, eliminate the samples of the ...
0
votes
0answers
13 views

Bourgain Embedding and Supervised Learning

I would like to know how to approach the following problem: Suppose we have a labeled dataset (+-1) and we do Bourgain Embedding of this dataset after defining some "reasonable" metric between two ...
1
vote
1answer
28 views

Comparing survey data to passively measured data

Basic Situation: I have quite a large sample of around 10k people. In that sample I have two values for each person: a passively measured value of an amount of time, as well as their estimation of ...
3
votes
1answer
72 views

Are there any mathematical features that an evaluation metric must have?

I'm trying to optimize the hyperparameters of my model using the Bayesian approach with the hyperopt library. I have to code a ...
0
votes
0answers
23 views

Weighted averaging of multi-task (multi-output) regression errors

I am trying to elaborate a multi-task (multi-output) regression metric based on single-task metrics. From my perspective, it should be a weighted average of single-task errors estimated with the same ...
0
votes
1answer
95 views

Higher RMSE but lower MAE and RMLSE. Which model is better? [duplicate]

I am evaluating two machine learning models. The output is count data which has a range of 0 to 30, which most of the output values being small values. Large output values are rare. One model has ...
0
votes
1answer
16 views

Distance metric for sequential spatial data (routes navigated in 2d space)

I'm looking for a distance metric to compute how close certain paths taken by people navigating throughout a city are to a set of 'correct' routes. I have path recordings for some 'correct' routes ...
0
votes
0answers
3 views

Evalution Metric for Recommender with one Relevant Document

Suppose I have a bunch of user session data. For each user session, 5 rows are created. Each row contains the user id, item id and whether or not they selected that item. For example : ...
1
vote
1answer
46 views

What do P and Q refer to in the Minkowski distance?

Wiki gives some explanation and a figure about the Minkowski distance: The Minkowski distance is a metric in a normed vector space which can be considered as a generalization of both the ...
0
votes
0answers
38 views

Prediction model with lagged target variable as input

Including a lagged version of the target variable as input naturally improves the accuracy. A disadvantage I observe is that almost all the weight (e.g. in linear regression) is put on that feature, ...
2
votes
0answers
57 views

Coming up with a metric that measures “homophily”?

[I'm not sure if this question goes here on Stats.SE --- please move if not.] Consider a 2D unit square. I observe N red and M blue balls on the square. I want to come up with a metric that measures ...
2
votes
2answers
48 views

Is there a metric for data where only true positives are labeled (no true negatives)?

Let's say I have a dataset where each item is labeled with either (1) true positive or (2) unknown (could be true positive, could be true negative). It seems like if there are only true positives ...
3
votes
1answer
131 views

How to optimize MAPE in regression algorithms

I have a regression task where the label is varying from about 0.001 to 1000. One of the feature called group, for example, group A corresponding label from 0-0.1 and group G corresponding label from ...
3
votes
0answers
67 views

Does this statistical distance have a name and does the triangle inequality hold?

Let $P$ and $Q$ be two distributions on $\{1,2,\dots,n\}$. Define their distance by $$d(P, Q)=\Pr_{X\sim P,Y\sim Q}[P(X)Q(Y)>P(Y)Q(X)]\,,$$ where $X$ and $Y$ are independent. I could show that $d(...
0
votes
0answers
25 views

Weighting for precision and recall

I want to integrate the notion of weighting into an evaluation. I am wondering if it is appropriate/correct to calculate precision and recall scores by adding a weighting on true positives, false ...
0
votes
0answers
61 views

Can NDCG be used a metric for evaluating implicit feedback based recommender systems?

If we are using binary feedback then Can NDCG be used a metric for evaluating implicit feedback based recommender systems
3
votes
1answer
46 views

Non-metrics give “pathological” solutions: what does this mean?

In this set of slides on DTW, slide 25 says that we generally prefer metrics over measures because, "Non-Metrics can sometimes give pathological solutions when clustering or classifying data etc." ...
0
votes
0answers
11 views

Metric for ranked keyword identification

I am trying to determine which metric(s) to use to evaluate the "coverage" of several lexicons (lists of words) with respect to a ranked list of significant keywords I have extracted from two ...
1
vote
0answers
73 views

How can I weight categorical variables to create a user preference score?

I'm working on a collaborative filtering algorithm, possibly paired with content-based similarity, for pairing users with other users. I have plenty of data on users and their like events of other ...
0
votes
2answers
44 views

Mahalanobis distance - vectors orientation?

In the Mahalanobis distance there are both $(\vec{x}-\vec{y})^T$ and $(\vec{x}-\vec{y})$. Which one is a column vector and which one is row vector? I need to write this in R, so vector orientation is ...
0
votes
1answer
130 views

Why is Pearson correlation is not an effective metric?

I found this statement in some documentation but I could not make sense of it. "Correlation is not a good metric for regression because it is scale and offset invariant". I understand that ...
0
votes
0answers
31 views

Set Similarity Measure that accounts for size

Suppose I have two sets, A and B where 10|A| = |B| and A is a subset of B. For the case where |A| = 1 the Jaccard similarity of the sets will be 0.1. For the case where |A| = 100 the Jaccard ...
0
votes
0answers
3 views

Comparing which histogram has overall low cost

Let's say there are two histograms which basically is constructed from array of numbers which is measured by, repeatedly performing a task by two different methods and individual numbers are time ...
0
votes
0answers
94 views

Is there a name for “normalized accuracy” as a statistic?

In short: I'm using a statistic representing the "normalized accuracy" of a confusion matrix. Is there a formal term for this? $$\text{normacc} = \frac{\text{acc}-\text{thacc}}{1-\text{thacc}}$$ ...
0
votes
0answers
205 views

Does Sørensen–Dice Coefficient (Dice Score) only account for true positives?

I'm working in a project on medical image segmentation which uses the Dice Score as part of the loss function, but I got some doubts with the commonly adopted implementation. The definition of Dice ...
1
vote
1answer
212 views

Difference between Euclidean ,Pearson, Geodesic and Mahalanobis distance metrics

Given a set of samples $X$. We are tasked to find an appropriate distance metric for $X$ from the given options which are Euclidean Pearson Geodesic and Mahalanobis distance metrics. To solve this, ...
1
vote
1answer
108 views

Clustering data with covariance for each point

I am looking to cluster data points that each have a covariance around itself (based on some function of its neighbourhood, but how I got it is not important). I would like to use the covariance to ...
0
votes
1answer
218 views

ROC-AUC score in sklearn

I'm trying to understand ROC-curve and AUC characteristic for it and found that behaviour of sklearn function roc_auc_score ...
0
votes
0answers
28 views

Is there a difference between Hausdorff Distance and Discrete Frechet Distance when working on time series?

I'm currently doing a little research on which kind of distance metric is the best for comparing the time-series I'm working with. To be clear, I'm doing this for a computer science internship, and I ...
0
votes
0answers
202 views

Why am I getting r2_score on test set as negative?

I wanted to initally test out without dropping any features (Redundant features such as ID are dropped). data_source -> https://archive.ics.uci.edu/ml/datasets/Automobile This was my procedure: 1) ...
0
votes
1answer
21 views

Creating a custom metric based on diverse feature scalse

I would like to create a custom metric that takes into account multiple features and produces a certain score. However, the issue is that I have data that has many different data types, such as ...
2
votes
0answers
47 views

Distance / Metric between two regression models? [closed]

I wonder if there is any theory or work about the "similarity" of two regression models. For example, if it is linear regression, the "similarity" could be defined by the l-2 distance between the ...
0
votes
0answers
35 views

Multiclass classifier with undefined prediction, how to calculate metrices

I build a multiclass classifier. I want the classifier to predict a few samples with little false positives, rather then many samples with lots of false positives. Therefore I want to choose a ...
0
votes
0answers
66 views

Metrics for comparing a multi-class model vs. a multi-label model?

The dataset is a multi-label dataset, where each item can have more than one labels. I first trained a multi-class classifier by randomly select the label for each item at each iteration, and ...