Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

0
votes
0answers
6 views

Should Learning Curves be Plotted only on Train or the Entire Dataset?

In order to compare a few models to start my ML project, first I split the dataset into train and test, and then performed nested CV on the training set only and got my fair estimate of true risk on ...
0
votes
0answers
47 views

Variable importance in a GBM

I have build a model with a Gradient Boosting Machine (GBM) and calculated the feature importance. All features are factors. Now, I know which features are most important. However, the features have ...
1
vote
0answers
10 views

High loss (low accuracy) on validation set but not on external test set

I'm training a neural network using 70% of my data as training set, 20% as external test set and 10% for validation using Keras. When I evaluate the trained model the performance on the validation set ...
0
votes
1answer
9 views

Evaluation network performance using cross-validation

Suppose I have a data set on which I'm training a neural network. I'm using four-fold validation, meaning that I train four models, one for each fold. Two of the folds are used for training, one for ...
0
votes
0answers
5 views

Model poor performance testing and solution

If my Model performance is good for training and validation set but testing performance ( on testing set ) is not good. Then what can be the reason and how we can resolve the issue ( improve the ...
1
vote
0answers
14 views

Comparing AUC between two subsets of test-fold

I suspect that the out-of-sample AUC of my model depends on the number of events in the out-of-sample/testing fold (I am modelling a binary variable, 0/1). In order to test this hypothesis, I want to ...
0
votes
0answers
7 views

Nested CV with Stratified Kfold

I am using nested CV for model evaluation and my target variable has imbalanced classes. With Sklearn, I am using GridSearchCV and cross_val_score to perform the nested cross validation. Each takes ...
0
votes
0answers
6 views

can I propgate machine learning lables in that way?

I have a golden data that I used to build prediction models and then I evaluate the model at the 20% of that golden data and the accuracy is almost excellent. Now, I am planning to use these ...
2
votes
0answers
31 views

Accuracy and ROC for Logistic and Decision Tree

So I run a logistic regression and decision tree model using same data. The accuracy shows that the decision tree outperforms logistic slightly. However, my ROC curve shows that logistic is much ...
1
vote
0answers
16 views

Test if generated image is considered as real image

I am planning my bachelor's thesis and I am going to write a new image generator. To evaluate it, I want to do an online questionnaire. I want to find out if people are able to differentiate between ...
0
votes
1answer
21 views

Statistically significant difference between real and dummy models

Suppose I'm building a machine learning model, like logistic regression, and I want to compare its performance with a dummy model, like taking the mean of the features in my training set. Suppose ...
0
votes
0answers
18 views

Do you need to adjust the probability if you use the 'class_weight' parameter in LogisticRegression-sklearn?

I have a imbalanced dataset and I want the the output as probabilities and not labels. Hence using Logistic Regression seemed to be the obvious choice. However the classsifer started predicting all ...
1
vote
1answer
8 views

Penalizing Some Errors more Than Others

Suppose you want to penalize a model when it makes mistakes in classifying some points in a test set more than other points in the test set. How would you do this? Would you just use another measure ...
0
votes
0answers
3 views

Precision metric over a subset of classes?

I have a classification problem where I want to measure the precision over only a subset of classes. That is, for each time the model predicts a certain class within the subset, what is it’s precision?...
0
votes
0answers
19 views

How to evaluate o model that predicts probabilities for a game between two players

I have a theoretical question. Let's say we have a game that is based on time (duration T) (something like football or basketball), and two players play each ...
0
votes
0answers
11 views

Classifier with good AUPR but bad AUROC

I am comparing a number of classifiers according to AUPR (area under the precision/recall curve) and AUROC (area under the ROC curve). There are some classifiers which are doing well (relative to the ...
0
votes
0answers
16 views

Metrics for comparing a multi-class model vs. a multi-label model?

The dataset is a multi-label dataset, where each item can have more than one labels. I first trained a multi-class classifier by randomly select the label for each item at each iteration, and ...
0
votes
0answers
17 views

Leave One Out Cross-Validation in Python

For me is not clear the way to implement LOOCV in Python, I have the next Python scripts: 1) ACC = 76.92 % ...
0
votes
0answers
20 views

How to choose metrics for evaluating classification results?

Recently we have developed a python library named PyCM specialized for analyzing multi-class confusion matrices. A parameter recommender system has been added in version 1.9 of this module in order ...
0
votes
0answers
24 views

Brier scores and integrated brier scores

Whats the difference between brier scores and integrated brier scores conceptually and mathematically? When would you use one over the other?
0
votes
0answers
11 views

Bayes R2 computation

I am working on evaluating the performance of a Bayesian network. One of the metrics I'm considering is the Bayes R-squared. On going through this publication, http://www.stat.columbia.edu/~gelman/...
0
votes
1answer
25 views

Interpretation of model

I have two models $Y$ ~ $X_1 + X_5$ and $Y$ ~ $X_2 + X_4$ ($Y$ is Binary). Both models produce different coefficients using training data, predicted probability using testing data and ROC curve using ...
1
vote
0answers
11 views

Rank error metric for time series

Suppose I have a collection of MAE across multiple time series (say, 10), and 3 models. However, MAE cannot be compared across time series. I compare the errors in this way: assign ranks to models, ...
0
votes
2answers
29 views

Splitting event data into training and testing with all the events in training ending before the events in testing starts

I have a dataset with event data. It has a date of start and date of finish vareiable. I need to predict time remaining until an event finishes. The problem is that I can't use events in future time ...
0
votes
0answers
37 views

How to interpret the Matthews Correlation Coefficient (MCC) for an imbalanced data set

I am trying to assess the performance of a machine learning model that has been passed down. The XGBoost model was trained on data that had a class imbalance of 84% majority class (label 0: 117,409 ...
0
votes
1answer
26 views

Retuning hyperparameters of the baseline when comparing it with a new model

I have a baseline model which has certain hyperparameters to tune (it's actually a neural network, but I don't know if it's important in this context). I want to compare it with my own extension of ...
2
votes
1answer
20 views

Cohen's Kappa - extreme disagreement doesn't register as strongly as extreme agreement

I'm wondering why, for the following data, the Kappa statistic wasn't $\pm 1$ but instead $-0.95$ for complete disagreement and $1$ for complete agreement. I expected them to have the same value. ...
0
votes
1answer
12 views

Are there any studies of generalized error performance that don't assume data quality is constant with sample size?

As far as I know, much of the statistical and machine learning literature where modeling algorithms are compared for their generalization error performance as a function of sample size (think of ...
0
votes
2answers
31 views

Which classification model should I choose and Why?

I am working on a research-based assignment where I suppose to build a 3-class (bad, medium, good) classification using SVM. The dataset provided is imbalanced. The train:test splitting ratio is 75:25 ...
0
votes
0answers
22 views

How much “ground truth” data do I need to evaluate a heuristic classification approach?

I have a heuristic approach to a de-duplication problem: There are about 300 million unknown data points. I attempt to match each data point with one of about 4 million independently known data ...
5
votes
0answers
20 views

How to verify if a prediction performance improvement is significant better?

I have a model M1 that achieved the predictive score (accuracy / AUC / F1 ...) of s1 in the test dataset. I developed a new ...
3
votes
1answer
26 views

Quantifying a manifold folding unto itself

I have a dataset of ~7k scattered points in 3D which represents a manifold that may or may not "fold unto itself". Here's an example where this does happen (look at the top-right yellow triangles): ...
0
votes
2answers
33 views

Evaluating Classifiers k fold CV or ROC

I've been doing a project to determine the 'best' classifier for classification on a dataset from UCI. I used 10 fold stratified cross validation to calculate the mean accuracy. However it was ...
1
vote
1answer
28 views

Suitable performance metric for an unbalanced multi-class classification problem?

I have an unbalanced multi-class classification problem with the following class distributions: Class 0: 17.1% Class 1: 63.2% Class 2: 19.7% I am using scikit-...
1
vote
0answers
13 views

Testing a new model [closed]

I've recently developed a new model for forecasting and the results seems good to me. I've used two different models with 5 different techniques involved in each model. The main motive here is to ...
1
vote
2answers
51 views

Why do we reserve a test set for final model evaluation?

Why do we reserve a test set for final model evaluation? Let's say you train a model following nested k-fold cross-validation and that you end up with one really good model among many that you tried ...
1
vote
1answer
33 views

Asymmetric error measure for forecasts

I am building a model for forecasting some number of activations. My data set has a panel structure. Now, I want to come up with a forecast performance measure to assess the performance of my model ...
0
votes
1answer
20 views

Do I include the validation set in final training?

For optimizing an unsupervised neural network with 1 hidden layer, I use the training set for training and the validation set for optimizing the number of neurons in the hidden layer (for example by ...
1
vote
0answers
15 views

Average time series forecast errors from cross-validation with rolling origin

I'm calculating the MAPE and RMSE over a rolling origin cross-validation with fixed forecast interval for several models. For example, for a daily series with 3 years, I'm training my model with 2 ...
0
votes
0answers
19 views

Evaluating binary classifier model. What can say precision, recall etc.? [duplicate]

i'm trying to understand wether my model has good performance or not. I have binary classifier for summarization sentences: important or not (extractive approach) on specific corpus. Dataset is ...
0
votes
0answers
13 views

Understanding lift curves

I understand the the cumulative gains chart shows the percentage of found targets (y-axis) (so the sensitivity) against the number of tested observations (x-axis). For the lift chart it seems that ...
0
votes
0answers
13 views

How to choose the best parameter for the LINEX loss function?

I am using a LINEX loss function to evaluate my forecast. What procedure should I follow to find the best $\alpha$ parameter? LINEX function: $$L(e) = \exp(\alpha e) - \alpha e - 1$$ Where $e$ is the ...
5
votes
3answers
827 views

What is the precise definition of “performance” in machine learning?

In machine learning, people usually refer to the "performance of a model" or "performance of an optimizer". What is the exact definition of "performance"? What would be the "performance of an ...
1
vote
0answers
43 views

What is the integral of the False Positive Rate over the False Positive Rate, compared to the AUC?

In machine learning the Area Under the Receiver Operating Characteristic Curve ($AUC$) can be illustrated in a plot of the True Positive Rate ($TPR$) against the False Positive Rate ($FPR$). Formally, ...
0
votes
0answers
45 views

Are Micro and Macro F1 enough for imbalanced classifier evaluation? What about AUC?

I'm working on an imbalanced classification problem. In the experiments, Micro-F1 and Macro-F1 are used for evaluation? But I can't get why the AUC score is not chosen for evaluation. Are these two ...
0
votes
2answers
23 views

When do we need cross validation? It's a lack of training data or choose different models?

When do we need cross validation? It's a lack of training data or choose different models? What is the background of the cross validation? What is the target of the cross validation?
0
votes
0answers
12 views

Analysing features of several classifiers

I am currently working on a small sentiment related project and need some advice regarding the evaluation. I trained different classifiers (Naive bayes, SVM with RBF kernel, SVM with linear kernel) ...
0
votes
0answers
18 views

Evaluating multiclass decision tree as a probability function

I have a problem where I am trying to estimate the transition probability within a state-diagram. For example, estimating the chances of transitioning from a happy mood to one of 3 possibilities (...
0
votes
1answer
25 views

How to diagnose loss curve not converging?

I am trying to predict remaining useful life (RUL) from temporal data with multilayered LSTM and obtaining the following curve: Looks like after first several epochs performance stops to improve and ...
2
votes
2answers
94 views

Can I trust my random forest model with low sample size and unequal class distribution?

I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution. I am doing some explorative modeling by using 400 features to classify ...