Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Sample size requirements for evaluating a many-class classifier

I am working on defining the requirements for a test/evaluation dataset for a many-class classifier (n_classes ~ 1,000), and am working to address concerns about ensuring the statistical power of the ...
jf2qm's user avatar
  • 1
1 vote
1 answer
44 views

Comparing forecasts under overlapping rolling evaluation windows

Two models with differing assumptions each purport to provide a forecast for a vector of values 1 year in the future. Each model has been run at the start of each month for three years: Run Date ...
David's user avatar
  • 515
0 votes
0 answers
13 views

How to Evaluate a Single-Value Prediction for a 6-Month Period Against Historical Data?

I'm tackling a time-series forecasting issue with daily granularity, aiming to predict a single aggregate value that represents the total sum of incidents over a 6-month period. My approach involves ...
Amit S's user avatar
  • 37
5 votes
2 answers
154 views

Linear Regression with Only Categorical Features: Evaluating the Model

Big Idea: This might seem a bit rambly, but there is a unified theme: how good is my model, and can I trust the predictions it's giving me? Background: I am performing a linear regression (not ...
Adrian Keister's user avatar
5 votes
2 answers
154 views

Does Bootstrapping the Test Set Provide a Real Error Confidence Interval?

My question here is a specific example of what was discussed in part in the answers of Bootstrapping test set? . Suppose I train a model where I cannot mathematically derive a confidence interval for ...
Ryan Folks's user avatar
0 votes
0 answers
29 views

Model evaluation approach and How it affects the performance of the model

So the task iam working on is supervised video summarization where the model tries to predict if a video frame is important or no using its features and the labels as annotations of frame scores. ...
moha tech's user avatar
0 votes
0 answers
10 views

Sampling for model validation against a full dataset

I have two models for estimating impact of some intervention on some metric. They take some sample as an input and return a single value. Full sample has 170k observations, which are a representative ...
Jean Broc's user avatar
1 vote
1 answer
65 views

Using unsupervised methods prior to cross-validation when all unlabelled data is available

There is lots of discussion about pre-processing methods and if they need to be included within a cross-validation procedure or if they can happen prior to splitting the data -- questions on ...
A. Bollans's user avatar
0 votes
0 answers
18 views

How to calculate AUC for a P-R curve with unusual starting point

I am working with a binary classifier that is outputting scores between 0 and 1, indicating probabilities of class membership, according to the model. I produced a P-R curve and the first point (i.e., ...
CopyOfA's user avatar
  • 177
0 votes
1 answer
19 views

Which data subset should be used for interpretable machine learning (IML)? [duplicate]

In a machine learning workflow, we need to split the dataset into training and test sets. We train several candidate models (typically tuned with hyperparameter optimization) on the training set and ...
Tripartio's user avatar
  • 2,146
1 vote
0 answers
29 views

Root Mean Square Log Error (RMSLE) Interpretation

I would like to clarify my understanding of Root Mean Log Squared Error (RMSLE): $$\text{RMSLE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left [\ln \left ( \frac{y_{i}+1}{\hat{y_{i}}+1} \right ) \right ]^{2}}$$...
ta1992's user avatar
  • 11
0 votes
0 answers
28 views

Compare the intercept and y-intercept of the new model with the old model

If we have two simple linear models (old and new), is there a good way to evaluate whether the newly developed model is sufficiently different from the previous model? For example, let's assume there ...
S. Jeon's user avatar
  • 35
0 votes
0 answers
13 views

Using a Gain Curve to Evaluate Predictive Model and Determine Optimal Cutoff / Threshold / Decision Point for a Classification Problem

When building or evaluating a predictive model, we know that a ROC curve can be useful for identifying the optimal cutoff/threshold/decision point in a classification problem with a dichotomous ...
JamesFM's user avatar
0 votes
0 answers
22 views

F1 score mismatch with publication

I'm trying to reproduce the results of the baseline model from SEP28k paper but I struggle to get the details. Most strikingly, the F1 score for random prediction doesn't match the paper. Here are the ...
marekjg's user avatar
2 votes
1 answer
76 views

Classification metrics in regression: can an analogue to precision, for instance, make sense on a continuum?

If we have a true classifier, it can make sense to calculate measures of performance like accuracy, precision (positive predictive value), and recall (sensitivity). Each of these has something to do ...
Dave's user avatar
  • 60.9k
0 votes
0 answers
19 views

How to validate unsupervised anomaly detection in absence of ground truth?

I am currently working on an unsupervised anomaly detection project and facing a challenge regarding the validation of the model's performance due to the absence of ground truth labels. I am using ...
Camilo Piñón's user avatar
1 vote
0 answers
56 views

Test set creation for a rare category classifier

I want to make a classifier for a very rare category. The base rate in a random sample is about 0.01%, estimated from finding about 10 positive examples using a zero-shot classifier on 100,000 ...
Quarticle's user avatar
1 vote
1 answer
32 views

What is the most appropriate index for categorical data clustering?

I am trying to replicate a study published by Bai & Liang, 2022 which focuses on clustering purely categorical data which are mostly found in the UCI repository. In my experiment for K-Modes, I ...
Gerard's user avatar
  • 181
0 votes
0 answers
8 views

Why detections count is not equal to unique truth count in YOLOv4 test result report?

I trained model using YOLOv4 on GPU, CUDNN and openCV (python) with AlexeyAB\Darknet with multi-label on windows environment. These labels are 25 classes (from 0 to 24). Then I test the model and I ...
N.white's user avatar
0 votes
0 answers
7 views

How does the training set size affect the uncertainty (variance) of performance estimation?

I am reading this paper which discusses the factors that affect the uncertainty (variance) in the performance estimation of a learner. The authors say (p. 2, "The monotonicity of the learning ...
ado sar's user avatar
  • 462
0 votes
0 answers
24 views

Equivalent to Kappa or MCC that compares to baseline classifier?

Both Cohen's Kappa ($\kappa$) and Matthew's Correlation Coefficient (MCC) measure the improvement of a classifier compared to a random classifier. This means that, for a classifier with a confusion ...
cdalitz's user avatar
  • 5,002
0 votes
1 answer
39 views

Normality assumption n=48 [closed]

Regarding normality assumptions for model evaluation I have been told by my supervisor that it is not needed in the case of analyzing but is needed in forecasting only. i am looking for an explanation....
Am Ahmed's user avatar
0 votes
0 answers
10 views

Is it meaningful to compare within-group AUROC between groups?

I have a risk model that I want to evaluate on different (patient) groups in order to compare how well the model is working on each of them. The groups may differ in size, baseline / prevalence / ...
Eike P.'s user avatar
  • 2,898
1 vote
1 answer
27 views

Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)

Consider these two approaches to evaluating a classifiers performance: Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
another_student's user avatar
0 votes
0 answers
49 views

How does `sklearn.metrics.roc_curve` work without using model predictions? [duplicate]

I am trying to understand sklearn's function for computing the roc_curve. If I understand correctly, one needs the TPR and FPR to compute ROC. However, sklearn's function takes as input - ...
desert_ranger's user avatar
0 votes
1 answer
43 views

Is it valid to add more data to a training data set after evaluating on a test set?

I'm working on a machine learning project to find particular key points in images. To do this, I'm using a U-net like architecture and treating it as a regression problem to produce a heat-map of ...
ocharles's user avatar
  • 103
2 votes
0 answers
34 views

what's the model/data drift metric for time series forecasting?

Suppose that I have time series forecasting model, e.g. forecasting point of sales revenue in various economic scenarios represented by indicators such as inflation or interest rates. I build a model $...
Aksakal's user avatar
  • 60.9k
0 votes
0 answers
46 views

Accounting for occasionally correlated data in performance analysis

I'm doing a study in which a new diagnostic method is evaluated. Samples are taken from patients, both this new method and the current standard method are applied, and we calculate concordance between ...
ADdV's user avatar
  • 101
0 votes
1 answer
43 views

How to interpret a confusingly large discrepancy between True Negative Rate and the number of Overspecified Models Selected by LASSO

Here it the link to the GitHub Repository for this project. I am going to preface my question by saying that this problem of interpretation I have run into is in the context of me doing my part as a ...
Marlen's user avatar
  • 147
1 vote
1 answer
396 views

why are models compared over multiple datasets?

I cannot clearly understand why models are compared across multiple datasets. What practical problems do they aim to solve? Especially, in several papers, models are compared using datasets that are ...
a_burkley's user avatar
0 votes
0 answers
5 views

Modeling considerations when data spans different events (time) and exhibit a (relatively) low mean and high variance

I have weekday data ($n = 1551$) from the past 5 years (2019-2023) with attendance at a large restaurant. I am just getting started, and for each weekday I calculated the mean and the variance as per (...
OLGJ's user avatar
  • 317
4 votes
1 answer
302 views

Plots to judge data that is a bit asymmetric

I have data that goes from negative to positive. When plotted in an histogram it looks like this The data is the "error". If I have another set of data I want to learn how to judge which ...
KansaiRobot's user avatar
1 vote
1 answer
56 views

Histograms and other plots to judge and analyze data

I have a variable that changes over time. Let's call this variable "the error". I use a histogram to graph how many of intervals of error has occurred. For example Here you can see that a ...
KansaiRobot's user avatar
2 votes
1 answer
36 views

Is R-squared valid for regularized linear models?

I found that there has been extensive discussion on the invalidity of R-squared for nonlinear models according to its original definition based on the following mathematical analysis,. The variance in ...
Alex's user avatar
  • 83
0 votes
0 answers
354 views

How to calculate FID for a set with a small number of images?

I need to evaluate my generative model using FID (Fréchet inception distance). However, the dataset of real images that I have only contains 2719 examples. I've read that the authors of the metric ...
nietoperz21's user avatar
0 votes
0 answers
18 views

Hold-out Set -- Smallest Acceptable Size given somewhat rare event?

I am contemplating the appropriate percent train-test split (e.g. 90%-10%, 80%-20%) Note I am asking about model evaluation, not cross validation/model building. This is for creating a hold out set to ...
purple-blade's user avatar
0 votes
0 answers
24 views

How to choose k for MAP@K?

Scenario: We want to evaluate our recommender system, which recommends items to potential customers when visiting a product detail page. Here are actual relevant items: ...
etang's user avatar
  • 907
1 vote
1 answer
73 views

Should my RMSE be smaller than my standard deviation [closed]

If the Root Mean Squared Error is larger than the standard deviation of my dependent variable, does that mean that my model is inaccurate? What would be a good threshold for the proportion RMSE and SD ...
Gabriel De Oliveira Caetano's user avatar
2 votes
0 answers
149 views

Implementing Quantile Loss function

I have been reading about Quantile Regression and the Quantile Loss function, but I have to admit I am a bit lost as how to practically implement it. I would like to use it to calculate the prediction ...
umbe1987's user avatar
  • 267
1 vote
1 answer
145 views

Can I say that the relative root mean squared error is the averaged percentage error?

RMSE is an error metric in which the mean of the data minimizes its loss function: $\text{RMSE} = \sqrt{\frac{\sum_{t=1}^{n}(y_t - \hat{y_t})^2}{n}}$ But it gives ...
Guilherme Parreira's user avatar
6 votes
2 answers
689 views

Why use a scoring rule different from the loss function?

I guess my question is related to these ones: Choosing among proper scoring rules, The performance metric used in prediction is different from the objective function to train the model, but I'm still ...
Igor F.'s user avatar
  • 8,817
0 votes
0 answers
61 views

Some curiosity about dice coefficient calculation

In semantic segmentation task evaluation with the following properties : batch size of the test set : 4 shape of a target mask/predicted mask : (1, 512, 512) number of batches in the test set : 30 ...
Cork's user avatar
  • 3
3 votes
1 answer
113 views

Is it ok to select models using MASE and present the metric to client area as MAPE?

My question follow this one. Metrics such as MASE and MSE have better properties than MAPE. ...
Guilherme Parreira's user avatar
0 votes
0 answers
25 views

Implementation of spBLEU

I was looking for a way to compute statistics for evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does ...
Prithvi's user avatar
  • 21
0 votes
0 answers
12 views

How calculate importance ratio for continuous states?

I am trying to understand importance sampling estimators, in particular for off-policy evaluation in reinforcement learning. I am working with the definition: The IS estimator provides an unbiased ...
sandboxj's user avatar
  • 111
0 votes
0 answers
23 views

Effective number of samples (ENS) and evaluation metric choice

I have a question regarding the appropriate evaluation metric for my problem. I'm working on a classification problem with highly imbalanced classes. I've decided to employ ENS (effective number of ...
Arya513's user avatar
3 votes
1 answer
51 views

Mincer-Zarnowitz test with cointegrated time series

The Mincer-Zarnowitz test of forecast optimality regresses forecast errors $e_{t+h|t}$ on the forecasts $\hat y_{t+h|t}$, $$ e_{t+h|t} = \gamma_0 + \gamma_1\hat y_{t+h|t} + u_t \tag{1} $$ or in ...
Richard Hardy's user avatar
5 votes
2 answers
478 views

Deleting outliers prior to data splitting or only in the training set?

I'm working on a dataset with some outliers in the response variable which are actually natural results (not errors). I want to calibrate a model which could then be used to predict on populations ...
Renaud Bied-charreton's user avatar
0 votes
0 answers
28 views

How to perform likelihood ratio test for bayesian neural network?

I am building a Bayesian neural network with Poisson likelihood and 50 features for time series prediction. Parameters of the model are learned using variational inference. I am trying to see whether ...
newbie's user avatar
  • 225
2 votes
1 answer
70 views

Why does the mean AURPC go down the more examples one uses?

In the discussion of this question, the following new one arose: Why is the mean AURPRC higher the fewer examples are used? Here is a minimal (Python) code example showing the effect: ...
Tobias Hermann's user avatar

1
2 3 4 5
23