Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

0
votes
0answers
7 views

Is pooling acceptable to evaluate information extraction?

When dealing with information extraction of unbalanced classes (e.g. the desired class has a prevalence of 0.5%), the required sample size for validation might be huge (thousands of cases and more), ...
0
votes
1answer
34 views

Training error less than validation error, but higher than test error?

I have a time series regression prediction problem. So I divided the dataset into 3 parts: training (first 70% of the time series data) validation (from 70% to 85% of the time series data) test set (...
0
votes
0answers
14 views

Interpretation of a log-level regression in its associated 'level' form

Is it conventional to interpret a least-squares regression with a log-transformed dependent variable (log-linear model) in its "level" form? In other words, running a model with the outcome in 'log ...
0
votes
0answers
12 views

Best practice for presenting classifier accuracy using cross-validation?

I have a classifier whose classification accuracy I must present. Of course, I could just do e.g. a single 90/10 train/test split and report the test accuracy. However, my dataset is fairly small, so ...
0
votes
1answer
44 views

Should I Choose the best model based on test error or validation error?

I divided my dataset to training, validation and test sets. Then trained multiple forecasting models on the training dataset. now I have 3 errors for each model: Training error Validation error Test ...
1
vote
0answers
31 views

Questions about confidence interval and presenting metrics for classification problem

A friend of mine and I are working on our bachelor's thesis and we're getting close to its completion. We have a few disagreements about how to present the result metrics for the experiment that we're ...
0
votes
0answers
10 views

Compare and visualization of multiple output data files

I have 100 data files corresponding to 100 samples. For these 100 data files, I have processed data in 4 different ways to generate result output files. Out of the 4 results files per sample, I ...
2
votes
0answers
93 views

Leave-one-out cross validation in regression: R squared cannot be used - how else may model performance on unseen data be evaluated?

I have a regression problem with very few datapoints, therefore I want to use leave-one-out cross validation (effectively N-fold cross validation with N being the number of datapoints) to determine ...
0
votes
1answer
15 views

Corroborating a differnce in differences identification strategy

I read in Mostly harmless econometrics that a good way of testing whether a difference in differences is a good identification strategy is running this equation: where the first sums are post-...
1
vote
2answers
24 views

How to compare two classifications, visually?

I have two classifiers, and I was asked (for learning purposes) to compare their predictions, visually. The target value is a real value (sentiment) What could be a good way to compare them?
0
votes
1answer
37 views

Is it acceptable to test on 0.01% of the training data?

I'm doing a cross-corpus evaluation on text classification with a LinearSVM. I was wondering if it is acceptable to skew the training-testing split more than the usual 80-20% split. Specifically, I ...
0
votes
0answers
18 views

Cross Validation and Feature Selection with Chronological Split and Feature Preprocessing

I have a task with daily entries for which I need to do binary classification. Suppose I have 18 months of data and the model is refit every month. In addition I've got about 150 one-hot encoded ...
0
votes
0answers
20 views

Interpretation of a brier score

Suppose that multiple brier scores were computed for two models $A,B$ and the density of the scores plotted as Where the average of $A$ is less than $B$ What would the interpretation of this be? ...
0
votes
0answers
17 views

optimism error rate calculation

As posted here in the book Elements of Statistical Learning we drive to the following: $$ w = \frac{2}{N}\sum_1^Ncov(y_i, \hat{y_i}) $$ The derivation is based on the following properties $E_y E_{Y^...
0
votes
1answer
23 views

Mathematical Motivation of Splitting Into Training and Testing Sets

In Learning from Data course taught by Caltech Professor Yaser Abu-Mostafa the following notation is used to describe the in sample and out of sample errors. $E_{in}[h]=\dfrac{1}{N}\sum_{n=1}^Ne(h(...
0
votes
0answers
51 views

Model evaluation metric when interested only on class distribution on sets of samples

I need to evaluate a sentiment classifier applied on posts gathered from some blogs. I am not interested on which post gets which sentiment value (positive/negative) but solely on the percentage of ...
1
vote
1answer
43 views

Is it possible to have recall and precision of 0 while having an area under PR ~0.5?

As the title suggests, I am running a Random Forest classifier using Scala. To evaluate this classifier (and since I am handling highly imbalanced classes), I used the ...
4
votes
1answer
96 views

How to Compute the Brier Score for more than Two Classes

tl;dr How do I correctly compute the Brier score for more than two classes? I got confusing results with different approaches. Details below. As suggested to me in a comment to this question, I ...
3
votes
0answers
54 views

Is there any strategy for validating the result of a general comparison between several confusion matrices?

Disclaimer: Recently we have developed a python library named PyCM specialized for analyzing multi-class confusion matrices. A compare system has been added in <...
1
vote
0answers
54 views

Evaluating Unbalanced Multiclass Classifiers: Which Tests to Use? [closed]

I am looking for some comprehensive instructions and ideally out of the box solutions (ideally for python) for evaluating different classifiers (which are already trained) for a multiclass ...
0
votes
0answers
17 views

How to evaluate the performance of ranking methods of items with binary and multiple labels?

I hope that you're all fine. I have two different datasets: Dataset 1 has 10 samples: 4 labelled as 'positive' and 6 as 'negative'. Dataset 2 has 10 samples: 3 labelled as 'high', 4 labelled as '...
1
vote
1answer
32 views

Check if my auto.arima model is good

I am a self learner, and even I am new to R, I have some data and I am trying to do time series analysis in R. I first tried to do an auto.arima fit to the data. I would liek to check if the ...
1
vote
0answers
37 views

R-squared vs MSE, why the discrepancy?

I am carrying out a project where I am imputing missing data. I am trying to compare an imputed dataset with a baseline dataset by measuring MSE and R-squared. These metrics are measured by ...
0
votes
0answers
13 views

How to best evaluate the quality / precision of geo-spatial prediction?

I have developed an algorithm to predict locations of certain items in 2D-space. The input data consists of various fuzzy / blurred observations of those items. Therefore, I am not performing a binary ...
0
votes
0answers
29 views

Quality assessment for multiple imputation when joint distribution is not multivariate normal

I have a dataset with 100+ columns and 1000+ observations with significant amount (>60%) of data missing and fraction of missing data in individual columns varying from 10% to 90%. Data in none of the ...
1
vote
1answer
32 views

Help me interpret my VGG16 fine-tuning results

I have a binary classification problem where I'm trying to classify whether a given cell is cancerous or not. For this I decided to play around with VGG16 pre-trained model and simply remove the last ...
0
votes
0answers
29 views

How to estimate the bias/variance percentages?

ML models performance can be evaluated using different scoring metrics; MSE, accuracy, precision as well as, it can be evaluated using learning curves. I am wondering how overfitting/underfitting ...
0
votes
1answer
32 views

Predicting an interval by deep learning or other machine learning methods

I have a distribution built on an interval for example [v_min, v_max], given a good estimate on the interval, the performance of the model can be good. If the ...
0
votes
1answer
17 views

Should Learning Curves be Plotted only on Train or the Entire Dataset?

In order to compare a few models to start my ML project, first I split the dataset into train and test, and then performed nested CV on the training set only and got my fair estimate of true risk on ...
0
votes
0answers
97 views

Variable importance in a GBM

I have build a model with a Gradient Boosting Machine (GBM) and calculated the feature importance. All features are factors. Now, I know which features are most important. However, the features have ...
1
vote
0answers
42 views

High loss (low accuracy) on validation set but not on external test set

I'm training a neural network using 70% of my data as training set, 20% as external test set and 10% for validation using Keras. When I evaluate the trained model the performance on the validation set ...
0
votes
1answer
9 views

Evaluation network performance using cross-validation

Suppose I have a data set on which I'm training a neural network. I'm using four-fold validation, meaning that I train four models, one for each fold. Two of the folds are used for training, one for ...
0
votes
0answers
7 views

Model poor performance testing and solution

If my Model performance is good for training and validation set but testing performance ( on testing set ) is not good. Then what can be the reason and how we can resolve the issue ( improve the ...
1
vote
0answers
27 views

Comparing AUC between two subsets of test-fold

I suspect that the out-of-sample AUC of my model depends on the number of events in the out-of-sample/testing fold (I am modelling a binary variable, 0/1). In order to test this hypothesis, I want to ...
0
votes
0answers
29 views

Nested CV with Stratified Kfold

I am using nested CV for model evaluation and my target variable has imbalanced classes. With Sklearn, I am using GridSearchCV and cross_val_score to perform the nested cross validation. Each takes ...
0
votes
0answers
7 views

can I propgate machine learning lables in that way?

I have a golden data that I used to build prediction models and then I evaluate the model at the 20% of that golden data and the accuracy is almost excellent. Now, I am planning to use these ...
2
votes
0answers
54 views

Accuracy and ROC for Logistic and Decision Tree

So I run a logistic regression and decision tree model using same data. The accuracy shows that the decision tree outperforms logistic slightly. However, my ROC curve shows that logistic is much ...
0
votes
1answer
21 views

Statistically significant difference between real and dummy models

Suppose I'm building a machine learning model, like logistic regression, and I want to compare its performance with a dummy model, like taking the mean of the features in my training set. Suppose ...
0
votes
0answers
31 views

Do you need to adjust the probability if you use the 'class_weight' parameter in LogisticRegression-sklearn?

I have a imbalanced dataset and I want the the output as probabilities and not labels. Hence using Logistic Regression seemed to be the obvious choice. However the classsifer started predicting all ...
1
vote
1answer
8 views

Penalizing Some Errors more Than Others

Suppose you want to penalize a model when it makes mistakes in classifying some points in a test set more than other points in the test set. How would you do this? Would you just use another measure ...
0
votes
0answers
4 views

Precision metric over a subset of classes?

I have a classification problem where I want to measure the precision over only a subset of classes. That is, for each time the model predicts a certain class within the subset, what is it’s precision?...
0
votes
0answers
26 views

How to evaluate o model that predicts probabilities for a game between two players

I have a theoretical question. Let's say we have a game that is based on time (duration T) (something like football or basketball), and two players play each ...
0
votes
0answers
35 views

Classifier with good AUPR but bad AUROC

I am comparing a number of classifiers according to AUPR (area under the precision/recall curve) and AUROC (area under the ROC curve). There are some classifiers which are doing well (relative to the ...
0
votes
0answers
16 views

Metrics for comparing a multi-class model vs. a multi-label model?

The dataset is a multi-label dataset, where each item can have more than one labels. I first trained a multi-class classifier by randomly select the label for each item at each iteration, and ...
0
votes
0answers
177 views

Leave One Out Cross-Validation in Python

For me is not clear the way to implement LOOCV in Python, I have the next Python scripts: 1) ACC = 76.92 % ...
0
votes
0answers
30 views

How to choose metrics for evaluating classification results?

Recently we have developed a python library named PyCM specialized for analyzing multi-class confusion matrices. A parameter recommender system has been added in version 1.9 of this module in order ...
1
vote
0answers
32 views

Brier scores and integrated brier scores

Whats the difference between brier scores and integrated brier scores conceptually and mathematically? When would you use one over the other?
0
votes
0answers
18 views

Bayes R2 computation

I am working on evaluating the performance of a Bayesian network. One of the metrics I'm considering is the Bayes R-squared. On going through this publication, http://www.stat.columbia.edu/~gelman/...
0
votes
1answer
26 views

Interpretation of model

I have two models $Y$ ~ $X_1 + X_5$ and $Y$ ~ $X_2 + X_4$ ($Y$ is Binary). Both models produce different coefficients using training data, predicted probability using testing data and ROC curve using ...
1
vote
0answers
13 views

Rank error metric for time series

Suppose I have a collection of MAE across multiple time series (say, 10), and 3 models. However, MAE cannot be compared across time series. I compare the errors in this way: assign ranks to models, ...