Questions tagged [metric]

A metric is a function that outputs a distance between 2 elements of a set & meets certain strict criteria (some 'distance' functions are not metrics).

Filter by
Sorted by
Tagged with
0
votes
1answer
15 views

Distance metric for sequential spatial data (routes navigated in 2d space)

I'm looking for a distance metric to compute how close certain paths taken by people navigating throughout a city are to a set of 'correct' routes. I have path recordings for some 'correct' routes ...
0
votes
0answers
3 views

Evalution Metric for Recommender with one Relevant Document

Suppose I have a bunch of user session data. For each user session, 5 rows are created. Each row contains the user id, item id and whether or not they selected that item. For example : ...
1
vote
1answer
24 views

What do P and Q refer to in the Minkowski distance?

Wiki gives some explanation and a figure about the Minkowski distance: The Minkowski distance is a metric in a normed vector space which can be considered as a generalization of both the ...
0
votes
0answers
22 views

Prediction model with lagged target variable as input

Including a lagged version of the target variable as input naturally improves the accuracy. A disadvantage I observe is that almost all the weight (e.g. in linear regression) is put on that feature, ...
2
votes
0answers
56 views

Coming up with a metric that measures “homophily”?

[I'm not sure if this question goes here on Stats.SE --- please move if not.] Consider a 2D unit square. I observe N red and M blue balls on the square. I want to come up with a metric that measures ...
2
votes
2answers
41 views

Is there a metric for data where only true positives are labeled (no true negatives)?

Let's say I have a dataset where each item is labeled with either (1) true positive or (2) unknown (could be true positive, could be true negative). It seems like if there are only true positives ...
3
votes
1answer
38 views

How to optimize MAPE in regression algorithms

I have a regression task where the label is varying from about 0.001 to 1000. One of the feature called group, for example, group A corresponding label from 0-0.1 and group G corresponding label from ...
3
votes
0answers
55 views

Does this statistical distance have a name and does the triangle inequality hold?

Let $P$ and $Q$ be two distributions on $\{1,2,\dots,n\}$. Define their distance by $$d(P, Q)=\Pr_{X\sim P,Y\sim Q}[P(X)Q(Y)>P(Y)Q(X)]\,,$$ where $X$ and $Y$ are independent. I could show that $d(...
0
votes
0answers
16 views

Weighting for precision and recall

I want to integrate the notion of weighting into an evaluation. I am wondering if it is appropriate/correct to calculate precision and recall scores by adding a weighting on true positives, false ...
0
votes
0answers
20 views

Can NDCG be used a metric for evaluating implicit feedback based recommender systems?

If we are using binary feedback then Can NDCG be used a metric for evaluating implicit feedback based recommender systems
3
votes
1answer
46 views

Non-metrics give “pathological” solutions: what does this mean?

In this set of slides on DTW, slide 25 says that we generally prefer metrics over measures because, "Non-Metrics can sometimes give pathological solutions when clustering or classifying data etc." ...
0
votes
0answers
9 views

Metric for ranked keyword identification

I am trying to determine which metric(s) to use to evaluate the "coverage" of several lexicons (lists of words) with respect to a ranked list of significant keywords I have extracted from two ...
0
votes
0answers
29 views

How can I weight categorical variables to create a user preference score?

Sorry, I'm not a statistician, but trying to fake it until I make it for data science. I'm working on a collaborative filtering algorithm, possibly paired with content-based similarity, for pairing ...
0
votes
2answers
36 views

Mahalanobis distance - vectors orientation?

In the Mahalanobis distance there are both $(\vec{x}-\vec{y})^T$ and $(\vec{x}-\vec{y})$. Which one is a column vector and which one is row vector? I need to write this in R, so vector orientation is ...
0
votes
1answer
49 views

Why is Pearson correlation is not an effective metric?

I found this statement in some documentation but I could not make sense of it. "Correlation is not a good metric for regression because it is scale and offset invariant". I understand that ...
0
votes
0answers
19 views

Set Similarity Measure that accounts for size

Suppose I have two sets, A and B where 10|A| = |B| and A is a subset of B. For the case where |A| = 1 the Jaccard similarity of the sets will be 0.1. For the case where |A| = 100 the Jaccard ...
0
votes
0answers
3 views

Comparing which histogram has overall low cost

Let's say there are two histograms which basically is constructed from array of numbers which is measured by, repeatedly performing a task by two different methods and individual numbers are time ...
0
votes
0answers
43 views

Is there a name for “normalized accuracy” as a statistic?

In short: I'm using a statistic representing the "normalized accuracy" of a confusion matrix. Is there a formal term for this? $$\text{normacc} = \frac{\text{acc}-\text{thacc}}{1-\text{thacc}}$$ ...
0
votes
0answers
55 views

Does Sørensen–Dice Coefficient (Dice Score) only account for true positives?

I'm working in a project on medical image segmentation which uses the Dice Score as part of the loss function, but I got some doubts with the commonly adopted implementation. The definition of Dice ...
1
vote
1answer
57 views

Difference between Euclidean ,Pearson, Geodesic and Mahalanobis distance metrics

Given a set of samples $X$. We are tasked to find an appropriate distance metric for $X$ from the given options which are Euclidean Pearson Geodesic and Mahalanobis distance metrics. To solve this, ...
1
vote
1answer
32 views

Clustering data with covariance for each point

I am looking to cluster data points that each have a covariance around itself (based on some function of its neighbourhood, but how I got it is not important). I would like to use the covariance to ...
0
votes
1answer
64 views

ROC-AUC score in sklearn

I'm trying to understand ROC-curve and AUC characteristic for it and found that behaviour of sklearn function roc_auc_score ...
0
votes
0answers
19 views

Is there a difference between Hausdorff Distance and Discrete Frechet Distance when working on time series?

I'm currently doing a little research on which kind of distance metric is the best for comparing the time-series I'm working with. To be clear, I'm doing this for a computer science internship, and I ...
0
votes
0answers
85 views

Why am I getting r2_score on test set as negative?

I wanted to initally test out without dropping any features (Redundant features such as ID are dropped). data_source -> https://archive.ics.uci.edu/ml/datasets/Automobile This was my procedure: 1) ...
0
votes
1answer
21 views

Creating a custom metric based on diverse feature scalse

I would like to create a custom metric that takes into account multiple features and produces a certain score. However, the issue is that I have data that has many different data types, such as ...
2
votes
0answers
32 views

Distance / Metric between two regression models? [closed]

I wonder if there is any theory or work about the "similarity" of two regression models. For example, if it is linear regression, the "similarity" could be defined by the l-2 distance between the ...
0
votes
0answers
22 views

Multiclass classifier with undefined prediction, how to calculate metrices

I build a multiclass classifier. I want the classifier to predict a few samples with little false positives, rather then many samples with lots of false positives. Therefore I want to choose a ...
0
votes
0answers
27 views

Metrics for comparing a multi-class model vs. a multi-label model?

The dataset is a multi-label dataset, where each item can have more than one labels. I first trained a multi-class classifier by randomly select the label for each item at each iteration, and ...
0
votes
0answers
47 views

Computing and estimating the EER on an entire dataset

I have reproduced "Generalized End-To-End Loss For Speaker Verification". It describes a method to create a deep learning model that can derive an embedding (a vector of 256 float values) that ...
1
vote
0answers
69 views

How do you / can you compare Bayesian vs. frequentist regression models?

I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $...
0
votes
0answers
33 views

Replacement for angular distance metric

I am looking for a distance metric that could be used instead of cosine/angular distance for high dimensional data. Metric that is limited the same way as cosine/angular distance is would be great. ...
0
votes
0answers
22 views

Consistency metric explanation

I am trying to understand a bit more about the consistency metric (to understand how consistency-based subset evaluation works). I find on this paper the following equation : $$\text{Consistency}_s =...
1
vote
1answer
189 views

Suitable performance metric for an unbalanced multi-class classification problem?

I have an unbalanced multi-class classification problem with the following class distributions: Class 0: 17.1% Class 1: 63.2% Class 2: 19.7% I am using scikit-...
0
votes
0answers
18 views

Mahalanobis: Covariance or correlation? [duplicate]

Is the Mahalanobis distance using correlations or covariances (among two vectors) to determine the similarity? I know that both are quite the same and differ only by a division of a standard deviation ...
0
votes
0answers
10 views

Which performance metric to use for stratified data? [duplicate]

I'm trying to classify a data into 3 classes (supervised), one of which is heavily underrepresented in the data set. In order to combat this imbalance, I decided to stratify the data. Now I want to ...
1
vote
1answer
28 views

Is recall a relatively meaningless metric in a balanced dataset?

Just looking for a sanity check here. Leaving aside precision, is talking about the recall of a binary classification algorithm sensible where 50% of the cases presented to it are positive and 50% ...
0
votes
0answers
10 views

Machine learning: eval_function for multiclass classifcation

I have checked What are the measure for accuracy of multilabel data? but I think the discussion somehow confused between loss and evaluation function. Say I have a multi-class classification problem ...
1
vote
2answers
55 views

Metrics for unbalanced classes [duplicate]

I have been looking for good metrics on this data set I am working, however it is highly unbalanced. It has a total of 8 categorical classes, one of them is responsible for ~40% of data, another for ...
2
votes
2answers
38 views

Localized distance function on sequential binary data

I am trying to find a good distance function for sequential data that is all binary. For now, I am using Edit distance however I have some more domain-specific knowledge that I would like to ...
0
votes
0answers
24 views

Quality metric of sampled time series data

I have a time series that has too many points. I sample one in every 100 points, in order to reduce the amount of data I need to transmit from my measurement device. What accuracy metric can I use ...
1
vote
1answer
218 views

Imbalanced Test Data

I have an imbalanced (1:5) training and test set with only two classes and have oversampled the training set with SMOTE so that the class ratio is 1:1. The ML model gives values over 0.7 for accuracy, ...
0
votes
0answers
22 views

Measure for evaluating a generated ranking of items, when best k items are known

I am evaluating different kinds of feature selection algorithms. Some return a ranking of variables based on their usefulness (to predict a target variable, etc.). Lets say there are n variables in ...
2
votes
2answers
1k views

Loss function and evaluation metric

When building a learning algorithm we are looking to maximize a given evaluation metric (say accuracy), but the algorithm will try to optimize a different loss function during learning (say MSE/...
0
votes
0answers
102 views

Best performance metric for highly imbalanced dataset f1 score vs kappa vs AUROC

I have highly imbalanced data (like fraud detection). I usually use f1 score to evaluate model performance. But I also saw people recommend AUROC and cohen's kappa. I'm seeking expert opinion on what ...
0
votes
0answers
19 views

Correct way of calculating cohort-based daily retention metric

Background: I need to calculate retention of users observed in a system. I will use this metric to estimate the Bayesian probability of said system's retention levels in the future. This question is ...
0
votes
0answers
8 views

Does zero-padding truth and estimate deteriorate the NRMSE/NRMSD?

I feel like I am missing an important point, but maybe the NMRSE (or NMRSD) is just counter-intuitive. This is the equation that I use: $\textrm{NMRSE} = \frac{\sqrt{\sum_i \left( \textrm{est}_i - \...
1
vote
1answer
756 views

different trends in loss and AUC ROC metric

I am training a deep neural network for a binary classification I am using binary_crossentropy as loss and area under the roc curve as performance metric as ...
0
votes
0answers
20 views

is there a measure for the roughness of a contour plot

There has to be a measure for the difference between "instantaneous" change of "energy" along a line in a space compared to averaged changed of energy along a line. I could take a smooth surface in ...
1
vote
1answer
40 views

Can two imperfect rankings be combined to produce a better ranking?

So I've got a dataset that can be ranked in two noisy ways $R_1$ and $R_2$. Let's call $R_1$ and $R_2$ functions from the dataset $x \in R^N$ to a real number between 0 and 1. Since they are rankings ...
0
votes
1answer
228 views

Assess the dependence of LDA on the random seed

New to latent Dirichlet Allocation (LDA), I would like to be sure that my output (in the first the step, the word-per-topic probabilities) depends on the input merely, and is (somewhat) stable ...