A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Gains on test data set higher than that on training data set post balancing

I have an imbalanced data set (96-4 split between No and Yes cases). I am trying to build a decision tree model for classification after balancing my data set(tried different thresholds for ...
1
vote
1answer
32 views

How to compare and evaluate models for a new feature?

I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records. When I tried multiple models based on 6,7 and 8 features, I see the ...
1
vote
1answer
29 views

Can increasing the training data reduce bias

As per my understanding, there is high bias if the model is underfitted. Does the number of records in training data affects bias? I mean, if there is too less records in training data, can the model ...
0
votes
0answers
16 views

Evaluate precision and recall results

The following table shows the precision and recall values I obtained for three object detection models. I evaluate the first two models as the following. The target is to find the best object ...
0
votes
0answers
17 views

Comparing two tests: Diebold-Mariano vs. Giacomini-White

What is (are) the main difference(s) between the Diebold-Mariano and the Giacomini-White tests of superior predictive ability? When does (do) the difference(s) matter?
0
votes
0answers
40 views

validation of survival analysis model

I am currently looking for evaluating/validating a survival analysis model on quite highly right censored data set. The thing is that i have many individuals in the data set. I wanted to use c-index ...
1
vote
0answers
20 views

Evaluating model that is in production without access to new, labeled data

I have a deep learning image classification model that is in production. I am trying to get a sense of model drift by calculating distribution metrics (e.g. mean predicted probability per label) over ...
1
vote
1answer
43 views

Should I use the Diebold-Mariano test year-by-year or on the overall forecast?

I have built two models, one ARIMAX and one VAR, to compare against a baseline ARIMA model to predict a weekly economic time series of interest. I am primarily comparing the accuracy of my models ...
0
votes
0answers
15 views

What is accuracy rejection curve

I am trying to understand accuracy rejection curve from this paper . It defines ARC as "An accuracy rejection curve (ARC)is a function representing the accuracy of a classifier as a function of its ...
1
vote
0answers
23 views

How to evaluate state space model?

I have a state space model which is basically a Kalman filter. The parameters of the Kalman filter are unknown and are estimated from data using EM algorithm. After I get these parameters, what are ...
1
vote
0answers
22 views

When is accuracy a good metric (as opposed to precision, recall, F1)? [duplicate]

Suppose you have a perfectly balanced data-set. In which applications is accuracy a good metric? Are there applications where it's preferable to precision, recall, and F1 (all at the same time)?
0
votes
1answer
59 views

Can you do logistic regression in R on a full dataset rather than the training data (i.e. 80%) of the dataset?

Using a logistic regression model to predict something in my dataset but was just wondering do I need to split the full dataset into training data (i.e. 80%) and test data (i.e. 20%) to make a model ...
1
vote
1answer
22 views

how to interpret the case when the cross validation accuracy is more than the model accuracy

I've trained a ANN model which resulted in 94.62%, but when I do a 5 fold cross validation the mean accuracy is 94.75%. Also 4 out of 5 cross validated models accuracy is more than 94.62%. How to ...
3
votes
1answer
54 views

Counterfactuals in Econometric Modeling (Abortion-Crime Hypothesis Revisited)

Donohue and Levitt (2019) recently published a working paper revisiting the abortion-crime link. My question is specific to equation (2) in their paper (see below): $$ ln(CRIME_{st}) = \beta_{1}...
0
votes
0answers
20 views

Accuracy metric in LSTM not considers time offset for multivariate time-series classification?

So this is a kind of complex question, so I hope I formulate it good enough. I have a human activity detection task that binary classifies if a user does a specific action or not. For me, it is ...
0
votes
0answers
10 views

Performance measure for estimation for acoustic impulse response

In searching for a performance measure for assert the estimation quality of acoustic impulse responses. Ideal Acoustic Impulse Responses (AIRs) are usually modelled as trains of impulses: $$ h(t) = \...
0
votes
0answers
9 views

How can I compare the performance of two measurement modalities?

Let's say given some 3D images of the brain, I have two methodologies to measure the brain's volume. I'll call these two methodologies M1 ...
0
votes
0answers
10 views

The results of test set are better than train set. why? [duplicate]

I have made a comparison of logit, ddhazard, decision tree and random forest models on ROC curve. the results of test sample are better than train set. why?
0
votes
0answers
11 views

ensuring a fair evaluation of student performance using multiple evaluators

I have a question regarding how to ensure fair evaluation of people taking a test. The test is an interview scenario, and the trainee has to ensure they ask the correct questions in various categories ...
1
vote
0answers
43 views

How to calculate TPR and FPR and plot ROC curves for object detection?

According to its Wikipedia page, receiver operating curves are created by plotting the TPR vs. the FPR at various discrimination thresholds where: ...
0
votes
0answers
14 views

Confidence Intervals for the Classification Accuracy

I am developing a classification system and after some iterations I settled on a Random Forest algorithm as the final predictor. I would like to have the confidence intervals for the estimated model's ...
0
votes
0answers
15 views

Is it possible to evaluate too many models?

Even with nested cross-validation in which model selection is occurring on the inner loop, is it possible that the best model identified by the inner loop for testing on the outer loop is an overfit ...
1
vote
1answer
15 views

How to split training data when learning DNN for unknown test data?

I'm designing a CNN model for a data mining competition in which we are provided with N sample of training data. We do not know the test size, but presumably it is from the same distribution as ...
0
votes
1answer
7 views

How to compare log-loss across similar classification models with different baseline probabilities?

Suppose I have two datasets, A and B, which share a feature vector $X$ but have different units of analysis (e.g. people from two different countries). I have trained classifiers with the same model ...
0
votes
0answers
20 views

Using auroc score and brier score to evaluate the performance of fully parametric survival models

I developed both semi parametric and fully parametric multi state survival models in R and wondering how i can compare their performance. I saw similar question asked here and the answer was to use ...
0
votes
0answers
19 views

Comparison of ARIMA and VAR accuracy

Can someone help explaining how to compare a forecast from an ARIMA model and a VAR model. I have tried calculating MAPE, MSE, RMSE etc. for my VAR forecast, but i simply cannot get it to work. ...
0
votes
0answers
20 views

How high is multiclass AUROC too high?

Whenever I get AUROC above 80% for a binary classification problem I do my best to check for leakage and overfitting - and usually my intuition is right, true AUROC is closer to the 70%-75% range. ...
0
votes
0answers
3 views

Standard significance test to measure the significance of time saved due to use of a tool

I have a web tool which requires the users to perform annotation for a certain task. it takes about 55 seconds to complete one task in the interface. I have introduced an add-on which reduces the ...
0
votes
1answer
23 views

Model evaluation when training set has class labels but test set does not have class labels

Training set and test set are separated in 2 files. The training set has class label and Random forest, svm, and KNN can fit. However, the test set does not have class labels. How do you evaluate the ...
2
votes
0answers
27 views

How to check if data follows a specific distribution?

I have got xgboost model trained on binary classification problem. Scores are in continuous range from 0 to 1. How can I check if those scores comes from given distribution (e.g beta). Does it make ...
0
votes
0answers
17 views

Training and predicting a Decision Tree over the same dataset

I wanted to train and predict a Decision Tree over the same dataset because I supposed the metrics will be perfect (overfitting). So I took an imbalanced dataset which I wanted to use as training ...
0
votes
0answers
20 views

Forecast evaluation in ar model

I have to compare in R 3 autoregressive models I've previously identified and estimated.The comparison should be based on an out-of-sample prediction: using (T-R, in my case 216) observations to ...
3
votes
1answer
104 views

Cross Validation and Multiple Imputation for Missing Data

Using 10 fold CV for performance estimation of a logistic regression model, what is the appropriate way to incorporate multiple imputation for missingness across the predictors and outcome in which ...
1
vote
1answer
33 views

Compare different models via p-value, AIC and BIC

A sensor with the response Sw shall be investigated if it is affected by external influences like Temperature Tu and relative ...
0
votes
0answers
14 views

Unrealistically high AUC-ROC score comparing to control feature and other performance measures

I am making a binary classification using regularized logistic regression, with extreme unbalanced data. The target label is Tar and non-target label is ...
0
votes
0answers
65 views

Interpreting hamming loss for multilabel classification

I have a multi label - multi class classifier that aims to predict the top 3 selling products out of 11 possible for a given day. Using scikit learn's OneVSRest with XgBoost as an estimator, the ...
0
votes
0answers
8 views

Generative Adversarial Networks evaluation methods for one channel

So I'm currently studying GANs with a focus on CycleGAN. I have trained my network on simulated images and real images. I did not train them as pairs but I have pairs available. The idea is now to ...
0
votes
0answers
24 views

Precision-Recall Curve for imbalanced dataset and effect of swapping positives and negatives

We are currently trying to evaluate the performance of a binary prediction model, on a dataset which has a majority of positive samples. Having done some research, we read from this paper that pr-auc ...
0
votes
1answer
25 views

ROC score for binary classification problem, where the predictions are either 0 or 1

For problems with binary classification, roc auc curve or roc auc score is often used to rate a model. But does the ROC ACC make sense in the context of a binary classification model that outputs only ...
3
votes
0answers
52 views

How to explain random forest ML algorithm doesn't learn at all, while logistic regression learns very well?

My prediction task is as follows: Use name to predict people's ethnicity (into 4 categories: "English", "French", "Chinese", and "All others") as a multiclass classification problem. The name ...
1
vote
2answers
58 views

What is the most appropriate way to validate prediction models with clustered data?

I am attempting to develop and validate a multivariable classification model using data from 10 clinical trials. I would like guidance on the most appropriate way to validate (internally and ...
1
vote
0answers
21 views

Which are the typical metrics used to evaluate a model in Crossvalidation?

I have implemented a 5 fold Cross-Validation method for classifiers in MATLAB. To choose the best model i evaluate the best model as the one that obtained a better average accuracy. Now i would like ...
1
vote
3answers
35 views

Binary metric for an unknown, but highly imbalanced, data ratio?

I'm looking for a good metric to compare binary classification methods for a task where The data is highly imbalanced. The approximate data imbalance is unknown. There are certainly more than 100 ...
0
votes
1answer
24 views

Nested Cross-Validation vs. Split-Sample Validation With a High n:p Ratio

With a high sample:predictor (n:p) ratio, as opposed to nested CV, why not just go with the split sample approach in which CV is done on training data (e.g., 80%) for model selection and estimation of ...
1
vote
1answer
41 views

Nested Cross Validation - Which Models Should We Evaluate in the Outer Loop?

Lets assume for example that I am attempting to predict a binary outcome using p predictors in which n>p with methods including a LASSO Regression, a Logistic Regression and SVM with an RBF kernel. ...
1
vote
1answer
27 views

Nested Cross Validation - How to Improve Models Without Bias

With nested CV, how can we get a sense of how our model may perform in the outer loop before actually bringing it to the outer loop? Without nested CV, if I did simple 10-Fold CV on training data (...
2
votes
2answers
61 views

Bootstrapped confidence intervals for performance metrics of predictive models

This questions is inspired by “Are we confident our model’s recall is precise?” by Ron Itzikovitch. Suppose there is a labeled dataset $D$, and the goal is to build a predictive model. The dataset is ...
0
votes
1answer
46 views

How does size of test set affect the performance of a model?

My data set is divided into 80:20 train and test...i have performed 10 fold cross validation on the train data set and tested the 20 % dataset on each iteration ( so that test set is not touched while ...
1
vote
1answer
28 views

Log-Log regression and cost function

I have made a very siple linear regression model having used log-log tranformation for the y and one of the independent variables: log(y)=B0+log(X1)B1+X2B2 where B0 is the intercept and B1,B2 the ...
0
votes
0answers
6 views

What is the better of two strategies in measuring performance of k-fold cross-validation?

I am doing 10-fold cross-validation on 1400 instances. I am considering those two strategies: Concat 10 y_test vectors (each of 140 values) from particular folds and compute the measure (e.g. f1) on ...