Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

Filter by
Sorted by
Tagged with
-1
votes
1answer
23 views

Evaluate clustering accuracy based on an adjacency/similarity/connection matrix

Description In the classification tasks, the classification accuracy is computed by accuracy=n_correct/n_total For example, if I have three samples, and the ...
1
vote
0answers
20 views

Imbalanced class issue

I am taking my first steps in machine learning and data science area. I know for sure that my next task will be related to the imbalanced class problem. I’ve walked through many articles covering this ...
1
vote
1answer
10 views

Choosing best model produced from different algorithms. Metric produced by cross-validation on the train set or metric produced on the test set?

I know that choosing between models produced by one algorithm with different hyperparameters the metric for choosing the best one should be the cross-validation on train set. But what about choosing ...
0
votes
1answer
22 views

How to interpret the direct comparison of Continuous Rank Probability Score (CRPS) and Mean Absolute Error (MAE)?

Say I have a trained Random Forest (RF) consisted of $m$ decision trees and I am interested to estimate $y$ from $t_1$ to $t_n$. The good thing about RF is that I have an ensemble of estimators and a ...
5
votes
1answer
19 views

Good metric / method to evaluate balanced multiclass classification when some classes are more similar than others?

Surely I'm not the first person trying to do this, but can't find a good answer (probably because I'm not searching with the right terms). I have a problem with 10 balanced classes (0-9) where the ...
1
vote
0answers
11 views

Statistical benefits of K-fold cross validation

I understand how k-fold cross validation works. For each iteration, the training data is split into $k$ portions, and using $k-1$ portion of the data for training and the $k$-th portion of the data ...
0
votes
0answers
12 views

Understanding the approach of considering the 95% quantile value as the 95% Confidence interval upper limit

I'm confused from the statistical approach described below. I'm not from this field but I had to use this approach in my study. Any explanation would be great!. According to my understanding the ...
0
votes
0answers
7 views

How to select the weights in a TOPSIS analysis?

In TOPSIS the sum of all weights needs to be 1 and that the selection of weights usually falls on the decision maker. At first it seemed to me that there was room for considering the weights as ...
1
vote
1answer
42 views

Intuition behind Brier score weighing step for censored data

Sources seem to suggest that when calculating Brier scores involving right-censored data, one must weigh the otherwise mean square error function with the inverse probability of censoring weights ...
2
votes
1answer
28 views

What is “symmetry” in evaluation metrics

I'm seeing Mean absolute percentage error (MAPE) is not symmetric. Tried to understand what is symmetry here but didn't find a good answer online. Can I ask: What is symmetry in evaluation metrics? ...
0
votes
0answers
49 views

Model evaluation from a comparison between simulated density data and individual presence data

I built a mathematical model of range expansion for a mosquito species. The model output is a map (raster format) of simulated mosquito density (mosquitoes/km²) in a study area. Here is an overview ...
0
votes
1answer
29 views

What is the right machine learning research methodology to compare two neural network approaches against eachother?

I am new to machine learning research, and I have a general question regarding how to compare different models on the same data set. How does a top researcher do this, and what is expected from a ...
0
votes
0answers
14 views

Bias of False Positive Rate (FPR) estimator

When evaluating a classifier's false positive rate on a randomly sampled test set (of size much smaller than the population), is an estimator for the false positive rate considered a "ratio ...
0
votes
1answer
14 views

Modeling a target variable that is numerical discrete (with few possible values) but can't be treated as ordinal

I am currently trying to predict the outcomes of a search site for purchases. The three outcomes have value: 0 (no click) 1 (click and no purchase) 10 (click and purchase) It seems that the data is ...
9
votes
4answers
294 views

For hyperparameter tuning with cross validation, is it okay for the fold splits to be same for every hyperparameter trial?

For hyperparameter tuning (random search/ grid search/ bayesian optimization), there are many trials performed for each set of hyperparameters. To evaluate how good a set of hyperparameter is, we can ...
0
votes
0answers
15 views

XGB model (or any other ML model) objective function vs scoring metrics and log transformations of the target label

I spent some time googling and could not find a proper answer for my question, maybe I have some terms confused but here is the question: When fitting a XGB model (or any ML model like Keras ANN or ...
0
votes
1answer
35 views

What value for recall implies a logistic regression model is good?

I'm studying logistic regression using Python and about metrics to have a good model, I know this three: accuracy, precision and recall. In the same way, I was studying using a dataset about ads in ...
0
votes
1answer
26 views

Comparison of forecasting models at scale

I am working on a project and 225 time-series models were built. Their variations are related with the intervals in the data to be considered (1 year, 2 years or 9 years), the percentages of the ...
0
votes
0answers
11 views

roc auc for small class imbalance

I have a classification problem with class imbalance(1:6). I'd like to know if roc_auc is a valid metric for this level of imbalance. I know it's not good for severe imbalance, but what about a case ...
0
votes
0answers
11 views

Convert RMSE, MSE, AUC, AUC PR or log loss to accuracy

I'm using h2o automl system and it returns all the models evaluation scores in MSE, RMSE, area under curve(AUC), AUC PR (dunno what this is) and log loss. I need this in accuracy so I can compare it ...
0
votes
0answers
24 views

COCO evaluation - Negative values on AP and AR

I am trying to evaluate one of the models using COCO dataset metrics. For some of the precision metrics I am getting results that I can comprehend. For example:Average Precision (AP) @[ IoU=0.50:0.95 ...
0
votes
0answers
14 views

How to tune hyperparameter with imbalanced data

I am doing an hyperparameter tuning through GridSearchCV for a binary classification. ...
2
votes
1answer
53 views

When computing mAP for an object detection model, how many detections should one consider?

I am trying to write some code to evaluate the MS COCO style mAP (mean average precision, average computed at the category level) at different IOU levels in the context of object detection with a ...
1
vote
0answers
13 views

increase precision without hurting recall

I have a classification problem with class imbalance and after the oversampling, I get high recall ,accuracy and roc (around 0.85) while my precision and f1 is fairly low(0.50). I have used every kind ...
0
votes
0answers
27 views

When is a model “fit” when doing each iteration of learning curves by Andrew Ng?

'm doing an academic exercise on Andrew Ng's famous Learning Curves. These are the ones we get when you plot the Training and Cross Validation errors vs an increasing amount of samples. Instructions ...
2
votes
1answer
51 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
1
vote
0answers
60 views

How to computationally optimize ARIMA walk-forward validation?

Just fitted and ARIMA model model = ARIMA(Preco1, order=(6,1,2)) model_fit = model.fit(disp=0) In order to do the walk-forward validation of an ARIMA model on a ...
0
votes
1answer
22 views

Fluctuating Validation Loss and Accuracy while training Convolutional Neural Network

I am training a convolutional neural network with 3 layers to classify cancer cell images into one of the 2 classes. I am using ReLU activations to introduce non linearity and batchnorm / dropout per ...
1
vote
0answers
21 views

Testing whether $h$-step-ahead forecast errors are at most MA($h-1$)?

I am forecasting a weekly commodity price series. I use a rolling window for estimating my model, and from each window I make point forecasts for one, two and more steps ahead. I want to investigate ...
0
votes
0answers
7 views

What are metrics for evaluating recommender systems if most users only transacts once?

What are metrics for evaluating recommender systems if most users only transacts once? 80% only bought once, 10% twice, and so on If I use Precision@k=5, because most users are buying only once, then ...
0
votes
0answers
4 views

Selecting one among several similar performing classifiers

If I have several classifiers with no statistically significant difference in F1 score, then how to decide on choosing one of them? What measure can I look into in such case?
1
vote
1answer
48 views

What could cause facebook's Prophet model to do so poorly on these procedurally generated datasets, where one is a continuation of the other [closed]

Recently I've been looking into some easy out of the box modeling using Facebook's Prophet -- potentially to use in some projects at work. So far, I have been super impressed with everything that I've ...
0
votes
0answers
25 views

Is there a method for evaluating predictive models that accounts for difference between predicted/actual distributions?

I am using machine learning to predict the monthly productivity (in dollars) of various groups of people. My question relates to ways of measuring the performance of my model. The distribution of ...
1
vote
1answer
37 views

Is there a range of values in the Akaike information criterion (AIC) score that tells us that the model is correct?

I know that when choosing a model, the AIC and BIC criteria are considered since the one with the lowest value will be the one corresponding to the best model, however, I would like to know if; Is ...
0
votes
0answers
5 views

Best way to combine disparate cost vectors to a single cost score scalar

Suppose I have a system with four components each of which may occupy a certain state. Suppose that each state a component is in is associated with a cost vector (or scalar) representing the cost of ...
0
votes
0answers
12 views

Why accuracy over each k-fold cross-Validation differ alot? how can i improve that

I have a data set with 91 Obervsation with 700 features. I have reduced the dimension of data using PCA. Then I split the data-set into training and test with ratio= 70/30. After that applied kfold =...
0
votes
1answer
34 views

What is the-state-of-art for unsupervised Anomaly models through unlabeled data regarding evaluation/validation in 2020?

I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Since outlier detection is commonly considered ...
0
votes
0answers
14 views

Part-of-Speech tagging: what is the difference between known words and unknown words?

I am trying to understand the result evaluation table (table 1) of this paper. There are three different accuracies reported overall, ...
1
vote
0answers
9 views

Given a training data set for binary classification, can I estimate the optimal ROC AUC (or any other performance metric)?

Assume a labeled dataset $D$ with $n$ observations $X_1, ..., X_n$. Each $X_i$ is represented as an $m$-dimensional features vector: $X_i = [x_{i1}, ..., x_{im}]^\top$. The target labels are $y_i \in \...
0
votes
1answer
55 views

Peformance metric when not only accuracy is important but also standard deviation

To put in context, I'm using a Cox PH model in the area of Survival Analysis using lifelines package to predict when a customer will do something, if that even ...
0
votes
0answers
18 views

What does it mean for the RMSE to be the same as the mean of the range?

I made some predictions on some time series data. The plots look good, the predictions line up with the original values quite well. But the error values don't make much sense to me. I calculated the ...
0
votes
0answers
29 views

Two True Positive for one ground truth in object detection

I am wondering is it possible to have two true positive predictions for one bounding box ground truth only. Following this section from Stanford. They define truth positive like this: We start with ...
0
votes
0answers
16 views

Is there a “good” way to evaluate segmentation in weakly labeled image data?

I have an image dataset for anomaly detection, which has weakly labeled ground truth images for the anomalies. Therefore, if there is a defect in an image, the ground truth would have a relatively big ...
0
votes
1answer
27 views

Comparing classifier performance when using slightly different datasets

Let's say I'm trying to predict whether tomorrow's temperature is higher than today's based on historical data (2 time series A and B). I've chosen XGBoost for the task. For model selection (...
2
votes
1answer
39 views

Saving model with highest accuracy or with least loss?

I'm confused. I used to save the model with the best accuracy in validation. But now I'm wondering if this was wrong all the time and I should have saved the model with the least loss. On the other ...
3
votes
2answers
107 views

What is accepted practice for avoiding optimistic bias when selecting a model family after hyperparameter tuning?

This is an extension of a previous question: How to avoid overfitting bias when both hyperparameter tuning and model selecting? ...which provided some options for the question at hand, but now I would ...
0
votes
1answer
43 views

Measuring Cox PH predictions

I'm running a Cox PH model in python using lifelines package. The two performance measures this package offers is log-likelihood or concordance index. I am aware ...
1
vote
0answers
19 views

What metrics should I use when developing the first model?

When I build the first regression model for a new project, standard metrics such as MSE, MAE, R2 score can't tell if the first model will work well in production because there are no existing models ...
2
votes
1answer
60 views

Naive benchmarks for scoring rules

I am a non-mathematical R programmer who is completely new to the idea of scoring rules. I would like to start using them instead of classification evaluation measures like accuracy and recall, which ...
1
vote
0answers
40 views

Precision-Recall Curve and Area under Precision-Recall Curve (AUC)

I created model (logistic regression) and now trying to create Precision-Recall plot and calculate area under Precision-Recall Plot. I'd like to note that this model is defective: ...

1
2 3 4 5
17