# Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

994 questions
Filter by
Sorted by
Tagged with
16 views

### How to evaluate stacked classifiers

I have 2 sets of classifiers, both trained on 2 different feature sets extracted from the same data. I would like to combine them using the "stacking" method, which I understand as follows: ...
• 101
59 views

### Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?

$AUC=0.5$ and an ROC curve of a 45-degree line often are considered the baseline performance of a model, one that gets absolutely nothing from the features. If we predict the same (prior) probability ...
• 35.3k
13 views

### Overall acurracy +/- E (with 90% C.I.)

I am assessing the accuracy of my classification model. I performed a 4-folds cross-validation and I obtained the following OA = (0.910, 0.920, 0.880, 0.910). So, the average OA is 0.905. My dataset ...
• 113
6 views

### linear regression removing interce [duplicate]

I have 4 continuous x variables and it is a linear regression problem. I built the first model and recorded performance on the test data - Mean absolute % error. I also noticed that some x variables ...
• 205
23 views

### Evaluating a survival model when focus in on "real time"

Assume a pure prediction problem. Say I want to evaluate a prediction-focused survival model in the context of actual dates instead of at a specific survival time or integration thereof. What would be ...
• 137
1 vote
19 views

### Take into account uncertainty in ground-truth in a classification

I need to evaluate the accuracy of a prediction model (binary classification), given that I have some kind of uncertainty on the ground truth measurement. The model predicts the occurrence of an event ...
• 111
36 views

### Which metric(s) can I use to evaluate how well a group of binary models agree in their predictions?

My request is best explained with an example. Suppose the upcoming week of (American) football matches are ...
• 1,822
23 views

### ROC curve illustrates a perfect classificator however other metrics show worse values

how is it to explain that ROC illustrates such a perfect classificator however other metrics represent something different? The evalaution was done on CV of 5. ...
• 196
1 vote
29 views

### Does evaluation metric not matter for training a model? And why?

I think I'm misunderstanding something with training ML tree models. There's a bunch of evaluation metrics we can use. It seems like this metric however is only used after training, which does not ...
• 111
13 views

### Random performance of AUPRC

I've been trying to understand how to interpret what random performance would be for a model I have on the AUPRC score. By 'random performance' I mean the worst possible performance. Purely ...
• 141
1 vote
17 views

### Reporting performance measures for classification in percentage or fraction?

I have seen classification metrics like f1-score, precision and recall being reported both as fractions and percentages. These measures are between 0 and ...
1 vote
33 views

### ROC curve - Question with exercise

I'm studying the ROC curve theory but I'm struggling with an apparently simple exercise. To recap what I know: "The ROC curve is a graphical plot that illustrates the diagnostic ability of a ...
1 vote
40 views

### Why fewer number of principal components give better results in clustering?

I try to do an (unsupervised) clustering with sklearn, Python, by different algorithms (hierarchical, distance based, density based etc. ones). The data in question has few hundred original features ...
• 703
1 vote
25 views

### What kind of transformation should I do to improve my linear mixed model?

I had asked in a previous post how to check the validity conditions of a linear mixed model I made, and here is the link After additional diagnostics I have the impression that the distribution of the ...
• 559
15 views

### Guidelines for using NMS before calculating mAP for object detectors

I am having a hard time understanding how to use Non-Max Suppression (NMS) when trying to evaluate an object detection model, especially when paired with trying to calculate metrics like the mean ...
• 133
16 views

### How to find statistical significance of predictions made by one model in two different test sets

I have trained a support vector machine and a random forest classifier to make predictions in a certain period. Thereafter, I have excluded a certain period from the original test data to see how the ...
16 views

### When the accuracy curve is U-shaped

I am currently working on MLP-based recommendation system. During training, the model updates based on BCE loss function with train set, then shows the hit rate (rate of how ground truth item is in ...
• 63
8 views

### what does it mean if all my deep learning models have the same precision

I have 4 different deep learning models that have different accuracy different recall and f1 measure but have the same precision what could be the reason for that?
103 views

### Multiclass Unbalanced Classication :Very very low F1 scores, high precision low recalls

have three classes for sentiment (negative, neutral, and positive). I created synthetic fake data for the positive class the analogy now is 50% neutral, 45% positive, 5% negative. I get the metrics ...
46 views

### Best hyperparameter is not consistent among different seeds

I do hyper-parameter tuning on my network and it outperforms the simple classifier. The difference in classification is considerable after hyper-parameter tuning. But, the problem is that an optimal ...
• 33
54 views

### Calculating "accuracy", "recall" etc. without classification

I have a set of models, that I'm comparing to each other with respect to prediction of a binary event. I'm using a few proper scores (Brier, log), but I also need accuracy, recall, sensitivity etc., ...
32 views

### overfitting of random forest in r

I am running a random forest classifier in R and during 10-fold cross-validation, I discovered that the model is overfitting. I am using a grid search to find the best hyperparameters and used the ...
25 views

### Margins in a triplet loss function, should they also be used during validation?

I'm using a triplet loss model for a re-identification task and I'm using a margin of 1 during the training. During the evaluation, I am using the exact same loss function with this margin included. ...
1 vote
32 views

### Statistical significance of repeated cross validation

I'm building a risk model for predicting risk of event within a specific time horizon for some patients. I use ROC-AUC score to evaluate the model. My dataset is highly imbalanced (15 events within ...
21 views

### Deep learning model performance heavily dependent on initial model weights

While my deep learning model is quite robust to randomness coming from train/test data-splits, it is not robust to initial model weights. How should I go about this? Also, I'd greatly appreciate it if ...
21 views

### Gene classification dataset- lack of data

I have a gene dataset consisting of around 5000 genes out of which I only have information on around 450 genes (I cannot collect more information). Of the 400 genes, 308 are positive (positive set: &...
1 vote
18 views

### Is it disingenuous to test tree-based regressors on training data for the sake of comparison to a linear model?

I am currently performing an analysis in a domain that, in current literature, is predominantly populated by likelihood-based linear models (MLR, Poisson Regression, etc.). My intent is to implement ...
• 123
27 views

### Effectively evaluate a model with highly imbalanced and limited dataset

(This question was originally posted on the Data Science stack.) Motivation Most data imbalance questions on this stack have been asking How to learn a better model, but I tend to think one other ...
17 views

### How to evaluate the performance of recommender systems without having labeled data

I have a huge citation graph of research papers and datasets. So, there is an edge among two items when one of them cites another. So far I've used Node2Vec for creating a dataset recommender system ...
109 views

### What is the opposite of precision called?

I know that $$precision = {\text { true positives } \over \text { predicted positives}}$$ but what about ${\text { true negative} \over \text { predicted negative}}$? what is it called? Thanks
• 135
23 views

### Grid search with cross-validation performing worse on test set than baseline model

I'm building a LogisticRegression() model in scikit-learn. Using train_test_split(), I've split the data into ...
139 views

### Prediction metrics for left-truncated and right-censored data

I am trying to assess (out-of-sample) predictive performance of survival analysis models with left-truncated and right-censored data. Assume the training and test data, respectively, consist of ...
• 21
8 views

### How do I calculate the statistical power to find differences in the performance of the models to compare?

I am building an XG Boost model and I want to compare it with the results of a logistic model already built to predict hospitalization, however, I am asked to calculate the statistical power to find ...
• 1
13 views

### Decision Curve Analysis

I have a question about decision curve analysis. I have trouble understanding the common strategy of treat/intervention to all. I do not understand why the line does not extend towards the whole risk ...
8 views

### How can I evaluate my work if there is no benchmark study for my targeted domain?

I have some questions if possible... My project is to create a model to detect fake news in a specific domain, which has not been investigated in this specific domain by previous studies. Data on this ...
1 vote
20 views

### Reliably evaluate model performance with very few positive samples

I do a binary classification in the domain of predictive maintenance. Setup My dataset is highly imbalanced with only 17 samples of the positive class, but an nearly indefinite amount of negative ...
• 525
12 views

### What is the right way to compare non-learning based algorithms with learning-based algorithms?

Let's say that I apply an ML-based algorithm on a dataset, and using F-Score as the evaluation metric, obtain a score on the test set. Now, I have a non-learning based algorithm (say, a genetic ...
726 views

### Which metric to use to evaluate Quantile Regression?

I have a prediction problem for which I want to predict the 75% Quantile using Quantile Regression. I am a little bit confused on how to evaluate this model (and also compare different models). If I ...
• 301
1 vote
48 views

### What are "volatile" learning curves indicative of? [duplicate]

I have a dataset set with ~40 features onto which I'm applying a multi-layer perceptron for regression purposes. The train, validation, and test sets are made up of 3M, 800K, and 800K examples each, ...
• 585
28 views

### Metric: What metric should used to evaluate distance between probability vector?

For typically metric space, we got different metric uch as Euclidean distance $dist(x,y) = \sum{(x_i -y_i)^2}$ , L1-norm $dist(x,y) = \sum{|x_i-y_i|}$ to evaluate distance between two vector and ...
• 532
17 views

### Evaluate linear model - Need for test/train [duplicate]

I am not sure whether it is necessary to split the dataset before creating a regression model or not. I see many websites calculating R squared on the training data, while others say the data has to ...
• 109
63 views

### What is the points valuation for this problem (assuming that 5-pointer worth 1 point is standard)?

Given four teams in a tournament, they versus against all the other teams once for each to reach the best-possible place. Three points are awarded to the team winning a match, with no points awarded ...
16 views

### Assemble neural networks to improve performance [duplicate]

I am approaching the world of Geometric Deep Learning for the first time and I have a question, I hope someone can answer it. I am currently working on models to classify some drugs as highly active, ...
20 views

### How can I modify the mean absolute percentage error to account for the curve direction?

I am trying to predict the trend of a certain curve in the future (whether it will be increasing, decreasing, or remain constant). For evaluating my prediction, I am using the mean absolute percentage ...
42 views

### Performance evaluation with non-representative data

I am currently trying to apply some models for text classification on a binary task. The core two approaches that I follow are, on the one hand, using word2vec vector representations on a Random ...
• 43
331 views

### Analysis of forecast errors from Facebook Prophet

I created a forecasting model Facebook Prophet and now trying to analyse the forecast errors (yhat - forecasted). Following are 3 graphs I plotted First one is raw forecast errors, second one is ...
1 vote
15 views

### Can the MSE value determines if a model is more performant than another one?

I have to use the same training set and I have to find two models, one of the family ${𝑓(𝑥)=𝑎𝑥}$ and one of the family ${𝑓(𝑥)=𝑎𝑥+\frac{7}{10}}$. After found these models, can I say that the ...
1 vote
123 views

### Should 3-pointers be worth 2.5 points?

I was seeking an alternative scoring rule sets instead of three points for a win (gained more engaging and balanced) to cancel the theory "Banking a draw meaning a new kind of loss". It's ...
70 views

### Is "sensitivity at fixed specificity" a valid metric for comparing different classifiers?

For a given dataset, a common way to compare 2 classifiers is to compare their average validation accuracies using cross-validation. Is it valid to replace the accuracy with other classification ...
• 975