Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

How to evaluate stacked classifiers

I have 2 sets of classifiers, both trained on 2 different feature sets extracted from the same data. I would like to combine them using the "stacking" method, which I understand as follows: ...
  • 101
2 votes
2 answers
59 views

Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?

$AUC=0.5$ and an ROC curve of a 45-degree line often are considered the baseline performance of a model, one that gets absolutely nothing from the features. If we predict the same (prior) probability ...
  • 35.3k
0 votes
0 answers
13 views

Overall acurracy +/- E (with 90% C.I.)

I am assessing the accuracy of my classification model. I performed a 4-folds cross-validation and I obtained the following OA = (0.910, 0.920, 0.880, 0.910). So, the average OA is 0.905. My dataset ...
  • 113
0 votes
0 answers
6 views

linear regression removing interce [duplicate]

I have 4 continuous x variables and it is a linear regression problem. I built the first model and recorded performance on the test data - Mean absolute % error. I also noticed that some x variables ...
0 votes
1 answer
23 views

Evaluating a survival model when focus in on "real time"

Assume a pure prediction problem. Say I want to evaluate a prediction-focused survival model in the context of actual dates instead of at a specific survival time or integration thereof. What would be ...
  • 137
1 vote
0 answers
19 views

Take into account uncertainty in ground-truth in a classification

I need to evaluate the accuracy of a prediction model (binary classification), given that I have some kind of uncertainty on the ground truth measurement. The model predicts the occurrence of an event ...
  • 111
3 votes
1 answer
36 views

Which metric(s) can I use to evaluate how well a group of binary models agree in their predictions?

My request is best explained with an example. Suppose the upcoming week of (American) football matches are ...
  • 1,822
0 votes
1 answer
23 views

ROC curve illustrates a perfect classificator however other metrics show worse values

how is it to explain that ROC illustrates such a perfect classificator however other metrics represent something different? The evalaution was done on CV of 5. ...
  • 196
1 vote
0 answers
29 views

Does evaluation metric not matter for training a model? And why?

I think I'm misunderstanding something with training ML tree models. There's a bunch of evaluation metrics we can use. It seems like this metric however is only used after training, which does not ...
  • 111
2 votes
0 answers
13 views

Random performance of AUPRC

I've been trying to understand how to interpret what random performance would be for a model I have on the AUPRC score. By 'random performance' I mean the worst possible performance. Purely ...
  • 141
1 vote
1 answer
17 views

Reporting performance measures for classification in percentage or fraction?

I have seen classification metrics like f1-score, precision and recall being reported both as fractions and percentages. These measures are between 0 and ...
1 vote
0 answers
33 views

ROC curve - Question with exercise

I'm studying the ROC curve theory but I'm struggling with an apparently simple exercise. To recap what I know: "The ROC curve is a graphical plot that illustrates the diagnostic ability of a ...
1 vote
1 answer
40 views

Why fewer number of principal components give better results in clustering?

I try to do an (unsupervised) clustering with sklearn, Python, by different algorithms (hierarchical, distance based, density based etc. ones). The data in question has few hundred original features ...
  • 703
1 vote
0 answers
25 views

What kind of transformation should I do to improve my linear mixed model?

I had asked in a previous post how to check the validity conditions of a linear mixed model I made, and here is the link After additional diagnostics I have the impression that the distribution of the ...
0 votes
0 answers
15 views

Guidelines for using NMS before calculating mAP for object detectors

I am having a hard time understanding how to use Non-Max Suppression (NMS) when trying to evaluate an object detection model, especially when paired with trying to calculate metrics like the mean ...
  • 133
0 votes
0 answers
16 views

How to find statistical significance of predictions made by one model in two different test sets

I have trained a support vector machine and a random forest classifier to make predictions in a certain period. Thereafter, I have excluded a certain period from the original test data to see how the ...
0 votes
0 answers
16 views

When the accuracy curve is U-shaped

I am currently working on MLP-based recommendation system. During training, the model updates based on BCE loss function with train set, then shows the hit rate (rate of how ground truth item is in ...
  • 63
0 votes
0 answers
8 views

what does it mean if all my deep learning models have the same precision

I have 4 different deep learning models that have different accuracy different recall and f1 measure but have the same precision what could be the reason for that?
0 votes
1 answer
103 views

Multiclass Unbalanced Classication :Very very low F1 scores, high precision low recalls

have three classes for sentiment (negative, neutral, and positive). I created synthetic fake data for the positive class the analogy now is 50% neutral, 45% positive, 5% negative. I get the metrics ...
2 votes
1 answer
46 views

Best hyperparameter is not consistent among different seeds

I do hyper-parameter tuning on my network and it outperforms the simple classifier. The difference in classification is considerable after hyper-parameter tuning. But, the problem is that an optimal ...
2 votes
2 answers
54 views

Calculating "accuracy", "recall" etc. without classification

I have a set of models, that I'm comparing to each other with respect to prediction of a binary event. I'm using a few proper scores (Brier, log), but I also need accuracy, recall, sensitivity etc., ...
0 votes
2 answers
32 views

overfitting of random forest in r

I am running a random forest classifier in R and during 10-fold cross-validation, I discovered that the model is overfitting. I am using a grid search to find the best hyperparameters and used the ...
0 votes
0 answers
25 views

Margins in a triplet loss function, should they also be used during validation?

I'm using a triplet loss model for a re-identification task and I'm using a margin of 1 during the training. During the evaluation, I am using the exact same loss function with this margin included. ...
1 vote
0 answers
32 views

Statistical significance of repeated cross validation

I'm building a risk model for predicting risk of event within a specific time horizon for some patients. I use ROC-AUC score to evaluate the model. My dataset is highly imbalanced (15 events within ...
0 votes
0 answers
21 views

Deep learning model performance heavily dependent on initial model weights

While my deep learning model is quite robust to randomness coming from train/test data-splits, it is not robust to initial model weights. How should I go about this? Also, I'd greatly appreciate it if ...
0 votes
0 answers
21 views

Gene classification dataset- lack of data

I have a gene dataset consisting of around 5000 genes out of which I only have information on around 450 genes (I cannot collect more information). Of the 400 genes, 308 are positive (positive set: &...
1 vote
1 answer
18 views

Is it disingenuous to test tree-based regressors on training data for the sake of comparison to a linear model?

I am currently performing an analysis in a domain that, in current literature, is predominantly populated by likelihood-based linear models (MLR, Poisson Regression, etc.). My intent is to implement ...
  • 123
0 votes
0 answers
27 views

Effectively evaluate a model with highly imbalanced and limited dataset

(This question was originally posted on the Data Science stack.) Motivation Most data imbalance questions on this stack have been asking How to learn a better model, but I tend to think one other ...
0 votes
0 answers
17 views

How to evaluate the performance of recommender systems without having labeled data

I have a huge citation graph of research papers and datasets. So, there is an edge among two items when one of them cites another. So far I've used Node2Vec for creating a dataset recommender system ...
2 votes
1 answer
109 views

What is the opposite of precision called?

I know that $$precision = {\text { true positives } \over \text { predicted positives}}$$ but what about ${\text { true negative} \over \text { predicted negative}}$? what is it called? Thanks
  • 135
0 votes
1 answer
23 views

Grid search with cross-validation performing worse on test set than baseline model

I'm building a LogisticRegression() model in scikit-learn. Using train_test_split(), I've split the data into ...
2 votes
1 answer
139 views

Prediction metrics for left-truncated and right-censored data

I am trying to assess (out-of-sample) predictive performance of survival analysis models with left-truncated and right-censored data. Assume the training and test data, respectively, consist of ...
  • 21
0 votes
0 answers
8 views

How do I calculate the statistical power to find differences in the performance of the models to compare?

I am building an XG Boost model and I want to compare it with the results of a logistic model already built to predict hospitalization, however, I am asked to calculate the statistical power to find ...
  • 1
0 votes
0 answers
13 views

Decision Curve Analysis

I have a question about decision curve analysis. I have trouble understanding the common strategy of treat/intervention to all. I do not understand why the line does not extend towards the whole risk ...
0 votes
0 answers
8 views

How can I evaluate my work if there is no benchmark study for my targeted domain?

I have some questions if possible... My project is to create a model to detect fake news in a specific domain, which has not been investigated in this specific domain by previous studies. Data on this ...
1 vote
0 answers
20 views

Reliably evaluate model performance with very few positive samples

I do a binary classification in the domain of predictive maintenance. Setup My dataset is highly imbalanced with only 17 samples of the positive class, but an nearly indefinite amount of negative ...
  • 525
0 votes
0 answers
12 views

What is the right way to compare non-learning based algorithms with learning-based algorithms?

Let's say that I apply an ML-based algorithm on a dataset, and using F-Score as the evaluation metric, obtain a score on the test set. Now, I have a non-learning based algorithm (say, a genetic ...
8 votes
3 answers
726 views

Which metric to use to evaluate Quantile Regression?

I have a prediction problem for which I want to predict the 75% Quantile using Quantile Regression. I am a little bit confused on how to evaluate this model (and also compare different models). If I ...
  • 301
1 vote
1 answer
48 views

What are "volatile" learning curves indicative of? [duplicate]

I have a dataset set with ~40 features onto which I'm applying a multi-layer perceptron for regression purposes. The train, validation, and test sets are made up of 3M, 800K, and 800K examples each, ...
  • 585
0 votes
0 answers
28 views

Metric: What metric should used to evaluate distance between probability vector?

For typically metric space, we got different metric uch as Euclidean distance $ dist(x,y) = \sum{(x_i -y_i)^2} $ , L1-norm $ dist(x,y) = \sum{|x_i-y_i|} $ to evaluate distance between two vector and ...
  • 532
0 votes
0 answers
17 views

Evaluate linear model - Need for test/train [duplicate]

I am not sure whether it is necessary to split the dataset before creating a regression model or not. I see many websites calculating R squared on the training data, while others say the data has to ...
  • 109
0 votes
1 answer
63 views

What is the points valuation for this problem (assuming that 5-pointer worth 1 point is standard)?

Given four teams in a tournament, they versus against all the other teams once for each to reach the best-possible place. Three points are awarded to the team winning a match, with no points awarded ...
0 votes
0 answers
16 views

Assemble neural networks to improve performance [duplicate]

I am approaching the world of Geometric Deep Learning for the first time and I have a question, I hope someone can answer it. I am currently working on models to classify some drugs as highly active, ...
0 votes
0 answers
20 views

How can I modify the mean absolute percentage error to account for the curve direction?

I am trying to predict the trend of a certain curve in the future (whether it will be increasing, decreasing, or remain constant). For evaluating my prediction, I am using the mean absolute percentage ...
3 votes
2 answers
42 views

Performance evaluation with non-representative data

I am currently trying to apply some models for text classification on a binary task. The core two approaches that I follow are, on the one hand, using word2vec vector representations on a Random ...
  • 43
2 votes
2 answers
331 views

Analysis of forecast errors from Facebook Prophet

I created a forecasting model Facebook Prophet and now trying to analyse the forecast errors (yhat - forecasted). Following are 3 graphs I plotted First one is raw forecast errors, second one is ...
1 vote
0 answers
15 views

Can the MSE value determines if a model is more performant than another one?

I have to use the same training set and I have to find two models, one of the family ${𝑓(𝑥)=𝑎𝑥}$ and one of the family ${𝑓(𝑥)=𝑎𝑥+\frac{7}{10}}$. After found these models, can I say that the ...
1 vote
2 answers
123 views

Should 3-pointers be worth 2.5 points?

I was seeking an alternative scoring rule sets instead of three points for a win (gained more engaging and balanced) to cancel the theory "Banking a draw meaning a new kind of loss". It's ...
3 votes
1 answer
70 views

Is "sensitivity at fixed specificity" a valid metric for comparing different classifiers?

For a given dataset, a common way to compare 2 classifiers is to compare their average validation accuracies using cross-validation. Is it valid to replace the accuracy with other classification ...
  • 975
0 votes
0 answers
23 views

How to compare 2 classifiers using a classification metric?

Let's assume we have 2 binary classifiers (A and B) and some labeled dataset, and we want to compare A and B. Let's assume we use the ROC AUC as the metric (although it could be the accuracy or ...
  • 975

1
2 3 4 5
20