Skip to main content

Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Evaluating Dimensionality Reductions models that can only transform train data

To evaluate the performance of dimensionality reduction (DR) models like TSNE and TRIMAP what I do is the following: Fit a model on trained data and transform train data Fit a new model on validation ...
Stan's user avatar
  • 1
0 votes
1 answer
28 views

Compare how close two covariance matrices resemble each other

Say that my model assumes vector $v_t$ has covariance matrix $\Sigma$. i.e $E[v_t v_t^T] = \Sigma$. But the data I actually feed into the model has covariance matrix $\Omega$. i.e $E[ w_t w_t^T] = \...
Taylor Fang's user avatar
0 votes
0 answers
5 views

What are the best books about human rating and feedback systems?

I'm studying evaluation data collection and rating evaluation of LLM models using platforms of human raters like Surge or Scale AI. I'm also studying how we can use survey methods of users of LLMs to ...
Estimate the estimators's user avatar
5 votes
1 answer
332 views

What is the best epoch to evaluate the test images?

I created a training, a validation and a test set for an image classification task. Then, I did training using the training and did evaluation on validation set. So, the next step is to evaluate the ...
cancan's user avatar
  • 53
1 vote
0 answers
57 views

Model Evaluation Strategies for Wind Energy Prediction Using Forecast Wind Speed

I am working on a regression problem to predict the amount of wind energy generated at a specific location based on hourly data. The dataset comprises four years of observations, with one row per hour,...
Ric's user avatar
  • 11
0 votes
0 answers
22 views

Comparing performance of probabilistic regression models - how to adapt Brier score?

Suppose I have two predictions models, Model 1 and Model 2. I have a dataset containing observations, features and actual outcomes. For each observation, the “outcomes” (i.e. predictions) that the ...
Alex's user avatar
  • 101
0 votes
0 answers
9 views

What are the right metrics to validate the performance of a custom clustering model with three possible outcomes?

I have developed a custom clustering model on top of MiniBatchKmeans, that has three possible outcomes for each data point: Assign the point to the correct cluster. Assign the point to the wrong ...
Sanjay Mythili's user avatar
4 votes
2 answers
66 views

If R2 is not appropriate for non-linear ML algorithms such as Random Forests, can a Pearson or Spearman correlation be used as performance metric?

$R^2$ is not appropriate for non-linear models, such as Random Forest (RFs) models. https://arxiv.org/pdf/1611.03063 Is R-squared truly an invalid metric for non-linear models? https://...
JElder's user avatar
  • 1,037
1 vote
0 answers
43 views

Optimal performance measures for species distribution models with presence and pseudo-absence data

I am currently building species distribution models (SDMs) (or ecological niches) using machine learning algorithms to predict the potential spatial distributions of animal species based on ...
Pierre Levoisin's user avatar
1 vote
1 answer
24 views

Evaluating a classifier's performance on different groups of subjects

I have developed a binary classifier that predicts whether a subject is injured or healthy. I am interested to know whether my model performs better on certain groups of subjects than on others. For ...
Roy Phillips's user avatar
3 votes
2 answers
74 views

Can an Anomaly Detector be Tested with Data that it Labeled?

Is it wrong to leverage a model to label data, then perform a train/test split to evaluate the performance of said model? Assume I have an unlabeled data set where the missing labels are a binary ...
noNameTed's user avatar
  • 135
1 vote
1 answer
24 views

How to compare the performance of a linear regression model with a mixed effect model

I am analyzing data on glucose levels in mice between control and genetically modified (KO) mice. I have performed this experiment quite a few times (>5 independent experiments). For the analysis, ...
André Barros's user avatar
2 votes
1 answer
32 views

How to interpret the results of a classifier when train/test method gives much better results than cross validated one?

I need your help to understand a situation where using train and test set produces perfect results (in terms of accuracy, precision, and recall) but when cross validation is used, the accuracy on ...
letdatado's user avatar
  • 367
6 votes
2 answers
134 views

Is it possible to evaluate causal algorithms on real world observational data?

Lot of times I get asked to use causal algorithms (e.g. algorithms estimating intervention results, or in general causal inference algorithms) and to compare them against non-causal prediction ...
DaSim's user avatar
  • 460
0 votes
0 answers
30 views

Can you store the value of the predicted variable (Y) at each fold and then correlate the predicted values with the actual data?

In particular, imagine to have a set of features (X) that I use to predict a continuos variable (Y). Is it possible to use elastic net, in a cross-validation framework, use it to predict the value of ...
sup_use's user avatar
1 vote
1 answer
103 views

Choosing the correct evaluation metric between F1-score and Area under the Precision-Recall Curve (AUPRC)

We're currently working on detecting specific objects (e.g. poultry farms, hospitals) from satellite images. We've modeled the problem as a binary image classification task (i.e. classifying images ...
meraxes's user avatar
  • 739
2 votes
2 answers
47 views

Unreasonable estimate in mixed models with interaction terms

In a LMM with interaction effect, the estimate seems unreasonable because the Score suppose to range from 0-12. ...
hey0god's user avatar
  • 21
0 votes
0 answers
17 views

How to Train a Model on the Whole Outer-loop Training Set in Nested Cross-validation?

I'm implementing nested cross-validation for a machine learning project and need some clarity on the training process using the outer-loop training set. Here’s a summary of my process: Outer-loop ...
Surayuth Pintawong's user avatar
0 votes
0 answers
12 views

Should you compute the FID for the reconstructed images or for the image obtained from text?

I am trying to train a text to image model based on Muse. I am planning on how to evaluate the result and I realized there's something about the Frechet Inception Distance that I don't understand. ...
Wolfuryo's user avatar
  • 133
0 votes
0 answers
23 views

How boxplot over absolute error of learning models could be used to compare\evaluate learning models' performances?

Recently I crossed this paper which represents the evaluation of various models' performances within a single dataset by Boxplot over $Absolute~Error~(AE)$ as follows: Fig. 12: Boxplot of baseline ...
Mario's user avatar
  • 441
0 votes
0 answers
22 views

Edited nearest neighbor (ENN) under-sampling? When is that useful?

Tackling the problem of unequal group sizes, one would often intuitively think of increasing the number of observations in the minority group. But sometimes the opposite can also be useful, i.e. ...
Marlon Brando's user avatar
0 votes
0 answers
21 views

How to evaluate an Earth system model in light of the spatial variability of observed variables?

Context My effort is to evaluate the performance of a physics-based numerical model to determine how well it simulates different state variables of a soil column (1D inside the model). The temporal ...
Alireza Amani's user avatar
0 votes
0 answers
33 views

I screwed-up model selection but ended-up with a very good model; am I ok?

In a recent experiment, I made an oversight: I divided my data into training and testing sets and conducted cross-validation for model selection and hyperparameter tuning after having applied Boruta (...
Alek Fröhlich's user avatar
0 votes
0 answers
27 views

Why is my (Mistral) LLM (almost completely) stopping to learn on my synthetic data after the first epoch, yet not overfitting?

I am creating synthetic task-oriented dialogs that are rather complex. Training and validation losses suggest that the model (almost completely) stops learning, but does not start overfitting: How ...
DaveFar's user avatar
  • 93
0 votes
0 answers
17 views

Machine Learning Experiment: should training parameters be fixed for a valid comparisons across model?

I am training an autoencoder three times, each time on a different dataset. The three datasets all have the same number of features, but have vastly different sizes. Assuming one of the datasets is A ...
BovineScatologist's user avatar
2 votes
1 answer
53 views

On using the loss as a metric?

The context is model evaluation in supervised learning. I am coming from a numerical optimisation background. For me it is quite natural to use the loss of the model (what we optimise during training) ...
Lucas Morin's user avatar
  • 1,665
2 votes
1 answer
85 views

Which Forecast Evaluation Metric To Use?

It is a forecasting problem. I need an evaluation metric which penalizes under-predictions more than over-predictions. Also I want it's range in certain interval (say 0-100), so that it becomes easier ...
Shardul Pingale's user avatar
2 votes
2 answers
106 views

Can I skip test set and train on 100% of data?

Is it a viable solution to train on the whole dataset without splitting the data into 'train' and 'test' sets? In other words, is it okay to skip offline evaluation and only perform online evaluation (...
asparagus's user avatar
7 votes
1 answer
69 views

On unbiasedness of an optimal forecast

Diebold "Forecasting in Economics, Business, Finance and Beyond" (v. 1 August 2017) section 10.1 lists absolute standards for point forecasts, with the first one being unbiasedness: Optimal ...
Richard Hardy's user avatar
0 votes
0 answers
15 views

Metric for Bayesian ground truth recovery

I am developing a new Bayesian model and want to compare it to already existing Bayesian models with the same hyperparameters using a simulation study. I generated 50 datasets and fit 4 different ...
Marcello Zago's user avatar
0 votes
0 answers
15 views

Sample size requirements for evaluating a many-class classifier

I am working on defining the requirements for a test/evaluation dataset for a many-class classifier (n_classes ~ 1,000), and am working to address concerns about ensuring the statistical power of the ...
jf2qm's user avatar
  • 1
1 vote
1 answer
67 views

Comparing forecasts under overlapping rolling evaluation windows

Two models with differing assumptions each purport to provide a forecast for a vector of values 1 year in the future. Each model has been run at the start of each month for three years: Run Date ...
David's user avatar
  • 515
0 votes
0 answers
16 views

How to Evaluate a Single-Value Prediction for a 6-Month Period Against Historical Data?

I'm tackling a time-series forecasting issue with daily granularity, aiming to predict a single aggregate value that represents the total sum of incidents over a 6-month period. My approach involves ...
Amit S's user avatar
  • 57
8 votes
2 answers
225 views

Linear Regression with Only Categorical Features: Evaluating the Model

Big Idea: This might seem a bit rambly, but there is a unified theme: how good is my model, and can I trust the predictions it's giving me? Background: I am performing a linear regression (not ...
Adrian Keister's user avatar
5 votes
2 answers
202 views

Does Bootstrapping the Test Set Provide a Real Error Confidence Interval?

My question here is a specific example of what was discussed in part in the answers of Bootstrapping test set? . Suppose I train a model where I cannot mathematically derive a confidence interval for ...
Ryan Folks's user avatar
0 votes
0 answers
35 views

Model evaluation approach and How it affects the performance of the model

So the task iam working on is supervised video summarization where the model tries to predict if a video frame is important or no using its features and the labels as annotations of frame scores. ...
moha tech's user avatar
0 votes
0 answers
15 views

Sampling for model validation against a full dataset

I have two models for estimating impact of some intervention on some metric. They take some sample as an input and return a single value. Full sample has 170k observations, which are a representative ...
Jean Broc's user avatar
1 vote
1 answer
72 views

Using unsupervised methods prior to cross-validation when all unlabelled data is available

There is lots of discussion about pre-processing methods and if they need to be included within a cross-validation procedure or if they can happen prior to splitting the data -- questions on ...
A. Bollans's user avatar
0 votes
0 answers
28 views

How to calculate AUC for a P-R curve with unusual starting point

I am working with a binary classifier that is outputting scores between 0 and 1, indicating probabilities of class membership, according to the model. I produced a P-R curve and the first point (i.e., ...
CopyOfA's user avatar
  • 187
0 votes
1 answer
27 views

Which data subset should be used for interpretable machine learning (IML)? [duplicate]

In a machine learning workflow, we need to split the dataset into training and test sets. We train several candidate models (typically tuned with hyperparameter optimization) on the training set and ...
Tripartio's user avatar
  • 2,246
1 vote
0 answers
37 views

Root Mean Square Log Error (RMSLE) Interpretation

I would like to clarify my understanding of Root Mean Log Squared Error (RMSLE): $$\text{RMSLE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left [\ln \left ( \frac{y_{i}+1}{\hat{y_{i}}+1} \right ) \right ]^{2}}$$...
ta1992's user avatar
  • 11
0 votes
0 answers
28 views

Compare the intercept and y-intercept of the new model with the old model

If we have two simple linear models (old and new), is there a good way to evaluate whether the newly developed model is sufficiently different from the previous model? For example, let's assume there ...
S. Jeon's user avatar
  • 35
0 votes
0 answers
23 views

Using a Gain Curve to Evaluate Predictive Model and Determine Optimal Cutoff / Threshold / Decision Point for a Classification Problem

When building or evaluating a predictive model, we know that a ROC curve can be useful for identifying the optimal cutoff/threshold/decision point in a classification problem with a dichotomous ...
JamesFM's user avatar
0 votes
0 answers
22 views

F1 score mismatch with publication

I'm trying to reproduce the results of the baseline model from SEP28k paper but I struggle to get the details. Most strikingly, the F1 score for random prediction doesn't match the paper. Here are the ...
marekjg's user avatar
3 votes
1 answer
81 views

Classification metrics in regression: can an analogue to precision, for instance, make sense on a continuum?

If we have a true classifier, it can make sense to calculate measures of performance like accuracy, precision (positive predictive value), and recall (sensitivity). Each of these has something to do ...
Dave's user avatar
  • 65k
0 votes
0 answers
66 views

How to validate unsupervised anomaly detection in absence of ground truth?

I am currently working on an unsupervised anomaly detection project and facing a challenge regarding the validation of the model's performance due to the absence of ground truth labels. I am using ...
Camilo Piñón's user avatar
1 vote
0 answers
63 views

Test set creation for a rare category classifier

I want to make a classifier for a very rare category. The base rate in a random sample is about 0.01%, estimated from finding about 10 positive examples using a zero-shot classifier on 100,000 ...
Quarticle's user avatar
1 vote
1 answer
37 views

What is the most appropriate index for categorical data clustering?

I am trying to replicate a study published by Bai & Liang, 2022 which focuses on clustering purely categorical data which are mostly found in the UCI repository. In my experiment for K-Modes, I ...
Gerard's user avatar
  • 181
0 votes
0 answers
11 views

Why detections count is not equal to unique truth count in YOLOv4 test result report?

I trained model using YOLOv4 on GPU, CUDNN and openCV (python) with AlexeyAB\Darknet with multi-label on windows environment. These labels are 25 classes (from 0 to 24). Then I test the model and I ...
N.white's user avatar
0 votes
0 answers
14 views

How does the training set size affect the uncertainty (variance) of performance estimation?

I am reading this paper which discusses the factors that affect the uncertainty (variance) in the performance estimation of a learner. The authors say (p. 2, "The monotonicity of the learning ...
ado sar's user avatar
  • 477

1
2 3 4 5
23