Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

1
vote
0answers
8 views

Average time series forecast errors from cross-validation with rolling origin

I'm calculating the MAPE and RMSE over a rolling origin cross-validation with fixed forecast interval for several models. For example, for a daily series with 3 years, I'm training my model with 2 ...
0
votes
0answers
18 views

Evaluating binary classifier model. What can say precision, recall etc.? [duplicate]

i'm trying to understand wether my model has good performance or not. I have binary classifier for summarization sentences: important or not (extractive approach) on specific corpus. Dataset is ...
0
votes
0answers
11 views

Understanding lift curves

I understand the the cumulative gains chart shows the percentage of found targets (y-axis) (so the sensitivity) against the number of tested observations (x-axis). For the lift chart it seems that ...
0
votes
0answers
10 views

How to choose the best parameter for the LINEX loss function?

I am using a LINEX loss function to evaluate my forecast. What procedure should I follow to find the best $\alpha$ parameter? LINEX function: $$L(e) = \exp(\alpha e) - \alpha e - 1$$ Where $e$ is the ...
5
votes
3answers
806 views

What is the precise definition of “performance” in machine learning?

In machine learning, people usually refer to the "performance of a model" or "performance of an optimizer". What is the exact definition of "performance"? What would be the "performance of an ...
1
vote
0answers
23 views

What is the integral of the False Positive Rate over the False Positive Rate, compared to the AUC?

In machine learning the Area Under the Receiver Operating Characteristic Curve ($AUC$) can be illustrated in a plot of the True Positive Rate ($TPR$) against the False Positive Rate ($FPR$). Formally, ...
0
votes
0answers
9 views

Are Micro and Macro F1 enough for imbalanced classifier evaluation? What about AUC?

I'm working on an imbalanced classification problem. In the experiments, Micro-F1 and Macro-F1 are used for evaluation? But I can't get why the AUC score is not chosen for evaluation. Are these two ...
0
votes
2answers
20 views

When do we need cross validation? It's a lack of training data or choose different models?

When do we need cross validation? It's a lack of training data or choose different models? What is the background of the cross validation? What is the target of the cross validation?
0
votes
0answers
11 views

Analysing features of several classifiers

I am currently working on a small sentiment related project and need some advice regarding the evaluation. I trained different classifiers (Naive bayes, SVM with RBF kernel, SVM with linear kernel) ...
0
votes
0answers
9 views

Evaluating multiclass decision tree as a probability function

I have a problem where I am trying to estimate the transition probability within a state-diagram. For example, estimating the chances of transitioning from a happy mood to one of 3 possibilities (...
0
votes
1answer
21 views

How to diagnose loss curve not converging?

I am trying to predict remaining useful life (RUL) from temporal data with multilayered LSTM and obtaining the following curve: Looks like after first several epochs performance stops to improve and ...
2
votes
2answers
53 views

Can I trust my random forest model with low sample size and unequal class distribution?

I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution. I am doing some explorative modeling by using 400 features to classify ...
0
votes
0answers
13 views

How to evaluate the predicted output in terms of order

I am dealing with a pattern recognition problem, where I need to predict the order of N samples. The concept of classes does not exist here. More specifically, I am ...
0
votes
2answers
28 views

threshold cutoff value from ROC for test set evaluation, do I use the cutoff from test ROC or training ROC

Let's say I am doing logistic regression. I split my data into training and test. I get an ROC for my training data and it has a cut-off of 0.25 I calculate my evaluation metrics, let's say just ...
1
vote
2answers
38 views

Evaluation of Clustering method

I'm currently confused on choosing the method for evaluating different clustering techniques. From this paper, they followed the pipeline: use Hungarian assignment for matching the cluster with true ...
0
votes
1answer
23 views

The representation of F1-score on the Precision-Recall Curve

Is there a way to project the F1-score on the precision-recall curve for a such binary classifier? Is there a relationship between the area under the precision-recall curve and F1-score? ...
0
votes
0answers
21 views

Can a permutation test be used to evaluate a model?

Let's assume that we have used a given predictive model to generate predictions for an evaluation data set. Now I would need to use these predictions to decide how good is the model. Or, more ...
0
votes
0answers
38 views

How do we know that a model (really) has a predictive power?

Let's assume that we have a trained predictive model and some data set to validate / evaluate the model. We also have a measure of accuracy (mean squared deviation). We apply the given model to a ...
0
votes
0answers
33 views

How to deal with a negative Kappa in classification?

I have a dataset with one binary class to be predicted, with 18 binary predictors and 17400 rows. Here I used a stratified split, with approximately 85% (14648 rows) for training and 15% (2752) for ...
0
votes
0answers
11 views

Correlation of fitness vs performance on an ensemble model analysis

I have data in the form (the following are just dumb values I use for presentation purposes): ...
0
votes
1answer
51 views

How to determine if one predictive model is statistically significantly better than another one?

I have a data set, two competing predictive models (regressions) and I need to decide which predictive model is better. Let us also assume that I have a measure of accuracy (for example mean squared ...
2
votes
0answers
19 views

How to best communicate model improvements to non-technical stakeholders?

Working on a project to improve a model, and I want to be able to communicate to non-technical people how much the model has improved with recent work. Here are example numbers showing the improvement:...
0
votes
1answer
29 views

how to compare the performance of linear regression vs tree-based methods such as randomforest

When I have linear regression, negative binomial, ridge/lasso regression and randomforest, how can I compare their performances? I've read that between linear regression and ridge/lasso, one can ...
0
votes
0answers
8 views

Understanding the Gini/AUC metric as out-of-development performance metric

Assume we develop a model for a binary classification task that reaches a certain Gini/AUROC estimate on the validation ( or training ) sample, among others. This is an overall good metric, often used ...
0
votes
0answers
10 views

Select model based on multiple performance metrics

I have a few performance metrics - MAE, RMSE, and MAPE. I choose my model hyperparameters on the validation set using MAE so far. However, I compare models among themselves on the test set using all ...
0
votes
0answers
36 views

confusion matrix for one hot vectors

Let $\mathbf{Y}$ be a matrix where each row is a one-hot encoding of true labels and $\hat{\mathbf{Y}}$ be a similar matrix for predicted labels that are generated by a softmax function. How is the ...
1
vote
0answers
19 views

What is the FROC (free-response receiver operating characteristic) curve?

I've found it in this paper but can't find any publicly available definition of it. There, it is used to evaluate the performance of aggregate CNNs.
0
votes
0answers
13 views

Determine performance difference after software changes

Situation: For our software package, I want to check if there is performance degradation after software changes. Therefore we measure multiple operations and save the duration in milliseconds. There ...
0
votes
0answers
4 views

Measuring the performances of a classifier by incorporating the Precision and Recall of each class

I have a classifier with 3 possible classes, and I find the "accuracy" measure not very effective ($\frac{\#CorrectlyClassifiedSamples}{\#TotalTestedSamples}$) I then instead computed the $...
0
votes
0answers
9 views

Most suitable comparison between groups

I have a neural network based algorithm which it was trained with 5 different datasets (A, B, C, D and E). For each one, 10 training runs was made {(A1, A2, ..., A10), (B1, ..., B10), ...}. Then, it ...
1
vote
0answers
43 views

Evaluate the conditional variance forecast from a GARCH model

I wanna evaluate a simple GARCH(1,1) model for the conditional variance. Firstly, I understand that the conditional variance is unobserved and that is really the crux of the issue. Out-of-sample, I ...
1
vote
0answers
10 views

Random forest regressor has a negative score [duplicate]

I am using a RandomForestRegressor. When I check the score for the model with the training data, it's Rregressor.score(X_train,y_train) 0.8357837327169805 but when I check the score using the test ...
0
votes
1answer
18 views

Using summary statistics from full dataset for feature selection

I want to generate some summary statistics and look at the correlation between the variables of my dataset to remove certain features (very low variance, very high correlation). The dataset is the ...
2
votes
1answer
52 views

Machine learning algorithm test/evaluation sample size

I have recently implemented a machine learning algorithm as a part of a new credit risk scoring system. I would now like to evaluate the accuracy/performance of the algorithm when used in a "real ...
1
vote
1answer
15 views

Evaluate classifier based on predicted probabilities?

If I had a categorical response $Y$ and multiple categorical features $X$, and I wanted to fit a model to predict $Y$. If all I cared about was the eventual distribution of $Y$ (say in terms of %), I ...
0
votes
0answers
6 views

Difference between historical, performance and Observation window

I have taken up a logistic regression course on Udemy and the trainer is from India. He has talked about differences between historical, performance and Observation window. However, it is very ...
0
votes
0answers
19 views

How to evaluate the results of a multilabel classifier using the predicted probabilities?

I can use sklearn accuracy_score to evaluate de predicted values of my multilabel classifier. But how can I evaluate the predicted probabilities obtained with predict_proba?
0
votes
0answers
10 views

Why after multiple epoc the performance decrease on the train data?

I have a network and I notice that after multiple epoc the performance on the train data start decreasing. Why?
0
votes
0answers
25 views

Repeated CV evaluation with confidence intervals in R caret?

it occurs to me that there is a part of model evaluation that I have not understood yet. The problem that I am working on now illustrates the point well I think. I need to fit a model of >400 ...
0
votes
0answers
12 views

Use same probability threshold when evaluating various models

Suppose I've built n number of different binary classification models. For what it's worth, they all use the exact same input and output data for training, are evaluated on the same test and ...
2
votes
2answers
96 views

Why AUC is not a good performance metric for a classification model?

After understanding the benefits of AUC I was stumbled to know that in some scenarios it might not be a good performance metric for evaluating a classification model. The below are the 2 scenarios: ...
1
vote
1answer
51 views

More features, less F-Score

Is there any rule about relationship between number of features and performance of the model? Recently, I did an experiment on 3 sets of features (all extracted from a same dataset). The strange point ...
1
vote
1answer
13 views

Evaluating propensity score matches- what to do when ratio of variances or standardized means of difference go to infinity?

I am working on a project where I am comparing the effects of a particular treatment on patients with other patients who didn't receive the treatment. As I am trying to replicate a randomized ...
1
vote
2answers
104 views

Evaluate the performance of a model with bootstrap

This question is about the application of the bootstrap rule The population is to the sample as the sample is to the bootstrap samples.I have a small dataset about lung cancer.There are 160 patients ...
1
vote
0answers
55 views

How to evaluate performance of (variational) autoencoders?

Let's assume that you have trained your (variational) autoencoder on MNIST digits. After some time, you check the result and decide that the reconstruction is pretty good. But this is highly ...
0
votes
0answers
14 views

Mean Test Set Performance of LSTM & Evaluation

In the paper "Greff et al, 2017 - LSTM A Search Space Odyssey" they evaluate different variants of LSTM architectures against different tasks/datasets. Could you help me to understand the evaluation? ...
0
votes
0answers
7 views

Evaluate a method taking into account when it fails

I am performing a grid-search (looking all/many variations) of a hyperparameter X to find the optimum value of that parameter. Where X is a symmetric matrix, where I set different values in each ...
4
votes
1answer
164 views

Why is cross entropy not a common evaluation metric for model performance?

When we train a classifier, we use cross entropy as a loss function and, for example, an F-Score as an evaluation metric, but why? Why not use cross entropy on the test set to evaluate the model ...
2
votes
0answers
11 views

Validating performance of panel data based models

I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). ...
0
votes
2answers
40 views

Performance evaluation

I'd like to test the performance of a penalized regression. I did three separate regressions for each response variable (one numerical, one binomial and one multinomial). I was checking this link, and ...