Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

How to measure a deviation from the theoretical model in terms of both frequencies and probabilities

Consider for instance, that I perform an experiment that consists in observing events. Let there be $N$ different events in total. I have a theoretical model that tells me which events should appear ...
0
votes
0answers
7 views

Statistically test differences of model performance on two different datasets

Suppose I have an SVM and a NB classifier that try to predict ownership of a particular text. For each of the algorithms, I train a model on 10 disjoint sets of training data. Then, I test them on ...
0
votes
0answers
7 views

Can retraining predictive model solves Dataset shift?

Assuming we are using non-parametric models like gradient boosted tree, can retraining the model solves each of the dataset shift (1. covariate 2. prior probability 3. concept shift)?
0
votes
0answers
10 views

Pairwise external evaluation of clustering with a contingency table

I want to evaluate the clusters pairwise based on this publication: https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html (I have the ground truth of the real labels) So ...
1
vote
2answers
34 views

When is it okay to make changes to your model after validating?

Let’s say I’m building a model to predict cancer relapse for a scientific paper. I use my training set to build many models and validate the best one on my test set to get an AUC of 0.65. I then go ...
2
votes
0answers
34 views

Comparison of volatility models - using conditional standard deviations or conditional variances?

At the below content, I learned that; "an unbiased variance estimator's square root doesn't imply being an unbiased estimator of the standard deviation". Comparison of daily fitted volatility and ...
0
votes
1answer
19 views

What does it mean if Nemenyi test does not return that the performance of two methods are significantly different while the Wilcoxon test does?

I want to compare the performance of $N$ methods on $m$ datasets. I performed the Friedman test and after that the Nemenyi test. The Nemenyi test is though not powerful enough to conclude that there ...
0
votes
0answers
14 views

What if Nemenyi test cannot find any significant difference between the performance of two models but Wilcoxon test does? [closed]

I want to compare the performance of N methods on m datasets. I performed the Friedman test and after that the Nemenyi test. The Nemenyi test is though not powerful enough to conclude that there is ...
1
vote
1answer
11 views

Multi-class classification - confusing some classes is worse than others

I am quite a Rookie concerning machine learning and now thinking about the following problem: I want to assign a person to one out of 6 groups, let's say concerning their favorite colour --> works ...
1
vote
1answer
30 views

Can different models have the same accuracy?

I'm doing binary classification on different models, GLM, Random forest and SVM have the same accuracy, recall, specificity, precision and f1 score, however they all have a different AUC-PR curve. ...
0
votes
0answers
8 views

How to evaluate and report estimates of performance metrics for a nested cross validation process

I am developing a classifier for Human Activity Recognition Tasks. I am evaluating three different models to use for my purpose and from each one i would like to obtain an index of the performance in ...
0
votes
1answer
19 views

Cumulative Matching Characteristic (CMC) curve for multiclass setting

The CMC curve is supposed to be calculated for a gallery and a probe set, both sets of vectors identifying some person. What if I train a model on a closed set of say 10 people with 10 samples (e.g. ...
0
votes
0answers
5 views

How do I get comparable results for my 4 different networks?

I have implemented 4 different modifications to the U-Net given in Ronneberger et. al. (2018). All modifications seem to need different hyperparameters to converge on the same dataset (on a 3-fold ...
-1
votes
1answer
28 views

First Fold Performing Worse Than Other Folds in Cross Validation

I have done a 80:20 split on my data. On my training data, I am getting 88% accuracy, but I wanted to perform cross validation on my training set. Now my accuracy is being reduced to 80%, which is a ...
0
votes
0answers
13 views

How to interpret Hosmer and Lemeshow test in R?

I'm reading some researches about how to use Hosmer-Lemeshow Goodness of Fit (GOF) Test in R. The results are quite clear and reasonable: X-squared, Degree of Freedom, p-value,.. However when I take a ...
2
votes
1answer
12 views

Using a secondary classifier to evaluate a primary classifier?

Suppose I have trained an image classifier that predicts the prominent objects in an image and the output predictions of this model is displayed to end users of some application. After deployment (i....
0
votes
1answer
21 views

When is a dataset “too imbalanced” for AUC ROC and PR is preferred?

I’ve read that precision-recall (PR) curves are preferred over AUC-ROC curves when a dataset is imbalanced as there’s more of a focus on the model’s performance in correctly identifying the minority/...
1
vote
2answers
29 views

Why and when do we need to tune hyperparameters?

This might come as a basic question. But I need to understand why do we need to tune the hyper parameters in a machine learning model instead of going into a different model altogether. Or to put it ...
1
vote
0answers
24 views

How to judge whether model is overfitting or underfitting

When validation or test data cannot be predicted properly and the model is suitable only for train data, the model is overfitting or underfitting. When I looking for a lot of reference , it seems to ...
4
votes
1answer
75 views

Proper scoring rule when there is a decision to make (e.g. spam vs ham email)

Among others on here, Frank Harrell is adamant about using proper scoring rules to assess classifiers. This makes sense. If we have 500 $0$s with $P(1)\in[0.45, 0.49]$ and 500 $1$s with $P(1)\in[0.51, ...
1
vote
0answers
20 views

what performance metrics is more important to comapre classifiers? [closed]

I have classified my biological data by using a few machine learning algorithms and calculated sensitivity, specificity, AUC, accuracy, kappa, PPV and NPV etc.? which one of these metrics are the most ...
0
votes
0answers
13 views

Compare regression and classification models. Which fits better?

I have 2 seperate approaches to a similar problem, one approach is regression focused and the other is classification based. I have the same predictor variable data set but with different outcomes (...
0
votes
0answers
7 views

Benchmarking Regression Models

I am currently working on a prediction model using multiple regression and random forest as templates. I have settled on the evaluation Metrics MAE and RMSE. However, I am having trouble ...
0
votes
0answers
9 views

Can you have a concordance statistic (discrimination) when predicting a continuous outcome?

I'm writing a proposal for a prediction model predicting BP (continuous outcome, predicting trend over time). For assessing model performance, I'm seeing discrimination and calibration as the most ...
0
votes
1answer
9 views

Cumulative gains chart not accurately describing performance of model

I recently ran into an issue interpreting the results of a cumulative gains chart I had developed for some binary classification model. The model had a very high Area Under the Receiver Operating ...
0
votes
1answer
24 views

Weighted importance sampling (WIS) and Importance sampling (IS)

I am currently reading papers about off-policy evaluation (or counterfactual evaluation) of reinforcement learning policies, including ones about the doubly robust estimator. As in this paper https://...
0
votes
1answer
33 views

Which performance metrics for highly imbalanced multiclass dataset?

I have a dataset with 5 classes. About 98% of the dataset belong to class 5. Classes 1-4 share equally about 2% of the dataset. However, it is highly important, that classes 1-4 are correctly ...
1
vote
1answer
19 views

Cross Validation and model training

I am using sklearn to train two models and compare their outcome with each other but I am not sure how to evaluate the models. As I have little data (approx. 300 ...
2
votes
1answer
33 views

Reconciling differences in model performance across deciles – linear regression

I am performing a simple linear regression and have started to examine the performance of my model. One action I've taken is to stratify the dependent variable into deciles and summarize the ...
0
votes
0answers
5 views

Which test would be appropriate for estimating parameter robustnes of repeated experiments on randomized data subsets?

I have a text classification problem where my data is split into fixed training and testing parts, which can't be changed. In order to be able to compare my model's performance to existing work, and ...
0
votes
0answers
15 views

Performance metric for algorithm with multiple numeric targets

I am constructing an algorithm that predicts the weights of all ingredients in a food product. This is a linear programming algorithm that chooses ingredient weights such that the nutrient profile of ...
1
vote
0answers
19 views

Comparison of simple linear regression, stepwise, lasso, and ridge

I am a totally beginner of machine learning. Please understand if my question is somehow basic. :) I have a dataset of 25 features related to a rental house and I want to predict the price based on ...
0
votes
0answers
9 views

Improving a Deployed Production Machine Learning Model Using Failed Predictions

I'm curious how I can improve the performance of a deployed model using failed prediction data. It seems the convention right now is to deploy the model and evaluate its performance after a few months....
0
votes
0answers
35 views

5x2cv combined F test with hyperparameter tuning

Does anything speak against using the 5x2cv combined F test combined with k-fold cross-validation in order to compare two learning algorithms? So, on each of the 10 training sets, we first apply k-...
0
votes
0answers
5 views

Caret for model performing and tuning paramter

I would like to ask, if I could use cross validation in caret for tuning parametr lambda and also for evaluating model performance. If I use savePredictions="final" and also I will tune parametr, it ...
0
votes
0answers
13 views

What is a good metric for a binary classifier when we are more interested in Precision than Recall but care about confidence scores?

I have an heavily imbalanced dataset and am training a binary classifier (which produces scores in the range 0 to 1) and need a single summary metric to tune hyperparameters with. For my problem, ...
2
votes
1answer
31 views

Training data set and cross validation

I would like to have some clarification because I have the same doubt every time. Training data is used to build the model, Validation data is used with cross validation for hyper parameter ...
0
votes
1answer
18 views

What is the meaning of model selection and model assessement?

I think in model selection one has to look for the best tuning parameter. But what is the tuning parameter? Is this the k in the k-fold cross validation? And is the model assessment just the ...
0
votes
1answer
11 views

How to evaluate the perforamance of clustering model using python

I have implemented the k means clustering model using python , i would like to know whether my model is perfect or not , so that i want to know which performance metrics is used for clustering model ...
0
votes
0answers
58 views

Measuring performance of classifiers with different/extra classes

I'm not sure where to post this, or how best to explain, so please bear with the bullet point approach below! I have created a decision tree using "perfect" labelled data which works 100%. I have a ...
0
votes
1answer
41 views

I want to compare a forecast with actual inventory, what statistical tests can I use?

I have two datasets, both are .csv files: Forecast- Marketing team's forecast of inventory levels that would be required in 2019 Inventory - Factory's records of actual inventory values recorded in ...
0
votes
1answer
29 views

Finding optimal point in roc curve giving weighs to true positives and false negatives

I have a binary classification model whose ROC curve looks like the one below. The black point is the optimal probability threshold to use by calculating the geometric mean. However, that's a pretty ...
0
votes
0answers
11 views

Evaluating baggedETS on a Test set

I'm using the forecast package to test a variety of models on monthly sales for 400 products sold by my company. I'm following the practice of fitting on a test ...
0
votes
0answers
11 views

Level of Agree-ness in ensemble models

I am trying to analyze the output of multiple models (as in ensemble method), classifying a multiclass dataset. I would like to study the behavior of the models compared to each other when they are ...
0
votes
0answers
7 views

Low values of silhouette index

I get very low values of silhouette index (0.3) even when external evaluation values are perfect (ARI=1). What could be the reason? Only thing that I could think of, that is specific for this ...
0
votes
0answers
11 views

What small set of classification metrics are most complementary?

I'm working on a project where people can submit their own data and automatically run a binary/multi-class classification. Since I don't know how much the customer cares about True/False Positives or ...
2
votes
0answers
12 views

Linear predictive model: how to attribute performance to individual features?

Here is my use case: I am running some data analysis for a large retail chain who has thousands of outlets across the country. I am using a linear predictive model to predict whether we should start ...
0
votes
1answer
15 views

Forecast ARIMA and out of sample evaluation

I have a train daily data and test daily data. I fit an ARIMA to my train data, forecast it 7 days and I want to get some performance measures of the forecasted values. I can get performance measures ...
0
votes
0answers
18 views

ANOVA of Model Performance metric

I have conducted multiple simulations of a hydrological model under a matrix of scenarios and calculated a Nash-Sutcliffe Efficiency (NSE) value for each simulation. The NSE is a model performance ...
0
votes
1answer
18 views

How to evaluate/score STL Decompose algorithm?

I found there are many STL Decompose function available, such as: https://github.com/jrmontag/STLDecompose https://github.com/LeeDoYup/RobustSTL RobustSTL said it is better than tranditional STL ...

1
2 3 4 5
15