Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

Filter by
Sorted by
Tagged with
0
votes
1answer
12 views

Usage of tracking signal as a metric when number of forecasts is low

Suppose Tracking Signal (TS) is used as a metric to evaluate the quality of a forecast. Let a be the ground truth value and f be ...
0
votes
0answers
12 views

How to interpret Isolation Forest results on variations of train/test sets?

I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
0
votes
1answer
23 views

Common denominator between good/bad models for binary classification

I am working on a simple binary classification problem, where I compare various models. I have posted the mean accuracies for 5-fold cross validation for the models below (the area-under-curve values ...
0
votes
0answers
9 views

Dilemma on cross validation for evolving time series

Context: I am trying to use machine learning approach (RNN, CatBoost, etc. for example) to do 1-2 days ahead electricity price forecast. In particular, we are targeting extreme events of high price ...
1
vote
0answers
96 views

How to use statistics to speed up row-wise computations on a data.frame?

I have a data frame with 10,000 rows and 40 columns. I am trying to apply a function to each of these rows. For each row, I am expecting to return a scalar which is the value of the statistic I am ...
1
vote
3answers
58 views

How do I know if the difference in performance between two machine learning algorithms is significant?

I built two machine learning models, A and B, and performed 10 times repeated 10-fold cross validation (CV). The mean RMSE and $R^2$ (over all folds) for the models is: Model A: mean RMSE=0.780, mean $...
0
votes
0answers
35 views

How to explain the different components of Theil's U?

as I am working with time series a lot recently, my main objective is to automate as much as possible, when it comes to the coding part. Hence I am using Theil's U (new variant) to evaluate model ...
1
vote
0answers
29 views

Presence of underestimation bias in earnings predictions

I am working on a financial data that entails forecasted revenue a company generates over a fiscal quarter and the actual revenue for that quarter. ...
0
votes
0answers
5 views

Best way to take into account the number of labeled data to be used for semi-supervised classification task

I am having two datasets with features drawn from different distributions. Both datasets contain equal classes for the same task. The thing i am facing is called domain adaptation - so i train a ...
0
votes
1answer
25 views

What performance indices are best to compare two time series with different data length . Can you suggest a method to do the comparison in R/Origin

I have two data sets (observed and simulated). Observed data set is the snow depth observed at a location. The simulated is the model simulated snow depth data. These data sets have different lengths. ...
2
votes
0answers
97 views

How to calculate the evaluation metrics on streaming data for online ML algorithms

I am working on a binary classification problem where I need to develop an online ML model that can work on streaming data. However, I am not sure how can I use the evaluation metrics for ...
1
vote
0answers
13 views

How is Dowd's (2007) resampling procedure supposed to mitigate the problem of autocorrelated multiple-step-ahead forecasts?

Dowd "Validating multiple‐period density‐forecasting models" (2007) considers evaluation of multiple-step-ahead density forecasts. There is a know problem of dependence between forecast ...
3
votes
1answer
35 views

Why do we choose the hyperparameters that gives the lowest validation error? Do we assume that it also gives the lowest generalization error?

The usual way of selecting hyperparameters is to tune it on the validation set and select the hyperparameters that gives the lowest validation error (Lets assume the validation sample is large so we ...
0
votes
0answers
17 views

Differences in Model Fit Evaluation Between Exploratory Factor Analysis and Confirmatory Factor Analysis

For the same dataset, I tried both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). For evaluating EFA model, the cumulative explanation of all latent factors adds up to ...
1
vote
1answer
9 views

Gain of power by a smart choice of goodness of fit test

Suppose one would like to test that a sample of observations comes from Uniform(0,1) distribution. Instead of applying the Kolmogorov-Smirnov test on the sample, one may first apply the inverse CDF (...
0
votes
0answers
10 views

Evaluation metrics for out-of-fold predictions obtained through negative binomial regression

I'm formulating count data out-of-fold predictions using a negative binomial regression and i am a bit confused as to which evaluation metrics apply best. My dependent variable has a long tail ...
0
votes
0answers
13 views

How to properly report results for a medical image segmentation task?

Let’s consider a 2-class / binary segmentation problem where c=0 for background (healthy tissues) and c=1 for foreground (...
3
votes
2answers
65 views

Is standard deviation of the output layer probability vector a good proxy of model confidence?

I'm working on a multi-classification problem and I'm kinda new to neural networks, which is what I'm using. In the model I'm using now, the probability vector has to sum up to 1, which to me would ...
0
votes
1answer
34 views

How to find the optimum when using regularization?

Using regularization increases the training error and the validation possibly as well. How do I find the optimum? Still just the optimum of the bias² and variance, like here: Source: https://dziganto....
4
votes
0answers
35 views

Is it wrong to compare multiple models on the same test set and choose the best model?

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
1
vote
1answer
47 views

Measuring Precision/Recall on a biased sample

I am working with ML models that predict e.g. whether an email violates some corporate policy or not. In this case, the "positives" are emails that violate the policy, and the number of ...
1
vote
1answer
52 views

Why does the best model in the training set have the worst test result?

I have trained eight models using 10-fold cross-validation, and evaluated the models by using resampling technique as described here. The result shows that SVM with sigmoid kernel (SVM-s) and random ...
0
votes
0answers
8 views

Performance of random forest - differing results for MLeval and what to use?

I am building a random forest model using R and caret, doing cross-validation to tune mtry. This is within a Shiny app and I supply some input parameters first: ...
2
votes
0answers
50 views

Combine several performance metrics from several datasets

We are developing and evaluating a multi knee/elbow point detection algorithm. For our evaluation, we have 200 sequences of real data. These sequences were annotated manually. For each algorithm and ...
0
votes
1answer
53 views

In general, what are precision, recall, F1 that are reported in papers?

I used classification_report in sklearn library And, the picture below shows evaluation on my model (anomaly detector) In general, what are precision, recall, F1 ...
1
vote
0answers
11 views

Which performance metric can I use for (multi) aspect based sentiment analysis?

I'm working on creating a model that extracts and evaluates sentiments of aspects in the text. My problem is that I'm unsure how to evaluate my results. Currently, I'm looking at the sentiment and if ...
0
votes
0answers
13 views

Is evaluation of individual attributes fitted to prediction model possible in python?

I have been implementing this research work in python. This research work presented the evaluation of Random-Forrest, SVM and Linear Regression in form of IncMSE, RMSE, MAE and p value, for each ...
1
vote
1answer
15 views

Understanding multiclass log-loss

I'm trying to understand the multiclass log-loss as described in sk-learns documentation. The wording ' Let the true labels (Y) for a set of samples be encoded as a 1-of-K binary indicator matrix...', ...
0
votes
0answers
19 views

Cross-validation: matching folds when comparing two algorithms

I want to compare two algorithms using cross-validation on the same dataset. Should the data in each fold match across the two models? (i.e. training set of fold i of model 1 = training set of fold i ...
0
votes
1answer
34 views

Multi-armed Bandits

Could someone explains to me the notation of this function, I mean I understand that we take the average of sum of the rewards for some particular action, however the notation seems strange to me for ...
1
vote
1answer
33 views

Evaluating a multi-step forecasting model?

The literature is a bit confusing for me on this one, from what I understand, a great deal of papers evaluate multi-step forecasting models on a single forecasting horizon on the hold out set. It ...
2
votes
0answers
17 views

Can log loss be an evaluation metric for classification models?

I read several posts online about evaluation metrics for classification models. Only accuracy, precision, recall, F-1 score, ROC, AUC, Confusion matrix are mentioned. However, I found a couple of ...
1
vote
2answers
46 views

How can we assess Bias when estimating step functions?

I am currently conducting a simulation study in which there is a known true function $F(t)$ which is strictly decreasing and continuous over time. I want to know if a particular estimator $\hat{F}(t)$ ...
2
votes
1answer
17 views

Assessing the performance of a model that has a policy that makes it worse

Assuming that a churn model will be used to prevent customer churn using policies that encourage customers with higher churn score (higher likelihood of churn probability) to stay loyal to a company. ...
1
vote
0answers
29 views

Evaluation metric for imbalanced data

Hi I'm a CS graduate student I have a question for AI or data experts. I'm writing a paper My dataset is time-series sensor data and anomaly (positive class) ratio is between 5% and 6% you can see the ...
0
votes
0answers
18 views

Can dimensionality reduction improve regression model performance?

As in the title: does dimensionality reduction (DR) like PCA, or LDA can improve the performance of regression models like LinearRegression, SVR, or Neural Networks? By reducing the number of ...
0
votes
0answers
15 views

Evaluating a CNN -multi class model with two separate thresholds

I have a model that outputs three classes. But here instead of one threshold, it depends on a combination of two (user input threshold). One threshold varies from 0.1 to 1.0 and the other varies from ...
0
votes
0answers
18 views

How to interpret high and similar ROC-AUC across models with slightly more marked differences in prAUC? (multiclass classification)

I am trying to compare the performance of different models for a multiclass classification task. The dataset itself has about 50 different categories with a lot of imbalance (the cardinalities of the ...
0
votes
0answers
26 views

Comparing ROC AUPRC scores in case of different baselines

I have some imbalanced data for binary classification, which I have preprocessed in 2 different ways. That led to having a different number of observations and pos/neg ratio. Then I trained the same ...
0
votes
1answer
15 views

How to use data augmentation in the context of model evaluation in machine learning?

I'm trying to use data augmentation for training a model for a classification task. But I'm not sure about how to use data augmentation in a fair and meaningful way in the evaluation of a machine ...
1
vote
1answer
38 views

Evaluation metric for time-series anomaly detection

My dataset is time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ? When empirically ...
1
vote
0answers
8 views

If you want to train a model on the predictions of another trained model, would you use the same training set for both model?

Imagine I train a model and make predictions on it, denoting the training data by x_train and the predictions by y_pred_train. Then, I want to use those predictions in a second ML model. Would you use ...
0
votes
0answers
11 views

compare multi-classification models with different target length

I have a given classification task where I want to classify text based on top, and I also have a taxonomy of topics that looks like this: ...
0
votes
0answers
14 views

Evaluating a convolutional neural network on an imbalanced (academic) dataset

I have trained a posture analysis network to classify in a video of humans recorded in public places if there is a) shake-hand between two humans, b) Standing close together that their hands touch ...
4
votes
1answer
150 views

Result of a diagnostic test of a predictive model looking too good

I have created a predictive model that outputs a predictive density. I used 1000 rolling windows to estimate the model and predict one step ahead in each window. I collected the 1000 predictions and ...
0
votes
0answers
22 views

Underfitting - how to tell if model capacity is saturated or if model is difficult to train?

I have a dense CNN which is very slow to train and additionally exhibits unusual loss behaviour. This is illustrated by the graphs below: for the graphs with two curves blue is the training curve and ...
1
vote
1answer
29 views

Multiclass classification vs Binary classification with class merging: prediction accuracy

I have a dataset with 4 labels. For me the most important is to be able to distinguish label 1 from all other labels, I don't care that much about distinguishing between labels 2,3 and 4. The ...
3
votes
1answer
82 views

Assess calibration of a density forecast by Kolmogorov-Smirnov test on PIT of realized values

According to Elliott & Timmermann "Economic Forecasting" (2016) p. 429-430, Calibration requires that if a density forecast assigns a certain probability to an event, then the event ...
1
vote
1answer
43 views

How to compare clustering results between "raw" and normalized data

I have a dataset and I would like to apply a clustering algorithm to find some groups. I do not have any label, so it is just wondering if I can find relevant clusters. If it may help, it is ...
0
votes
1answer
29 views

How to assess feature usefulness?

I have a imbalanced dataset of 3k rows with 87:13 ratio of positive and negative classes. I am trying to do binary classification. Since my class proportion is skewed, I have optimize the decision ...

1
2 3 4 5
18