Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

Filter by
Sorted by
Tagged with
1 vote
0 answers
17 views

Balancing Multiple Evaluation Metrics for a Model

When evaluating a machine learning (or other statistical model) against multiple evaluation metrics, is there a standardized way to choose the "best" model? As a concrete example, for a two ...
user avatar
1 vote
0 answers
17 views

Interpreting learning curves

There is really few examples online regarding interpreting learning curves and they are all of different type.It is quite confusing to me honestly.May I just ask: How should we interpret them?What ...
user avatar
0 votes
0 answers
9 views

After training a model, how does test set error inform decision making?

I split a data set into three subsets: training, validation, and test sets. I use my training data for fitting and validation to check for overfitting. I then have a final model that I then propose to ...
user avatar
  • 1,003
2 votes
1 answer
59 views

Clustering while knowing the ground truth: Why would someone choose this approach?

If the ground truth of the class/cluster/segment that our observations belong to, is known in advance, why would someone choose to perform clustering instead of classification? In fact, doesn't the ...
user avatar
0 votes
0 answers
22 views

How to choose the best recommender system? What evaluation metrics to use?

I want to build a recommender system to suggest similar songs to continue a playlist (similar to what Spotify does by recommending similar songs at the end of a playlist). I want to build two models: ...
user avatar
  • 1
2 votes
0 answers
78 views
+250

How to evaluate complementary datasets for ML models?

Evaluating ML models is a fundamental task and subfield of the Machine Learning practice. On the other hand, I was not able to find any existing materials, guides, protocols, papers on how to proceed ...
user avatar
0 votes
0 answers
27 views

Random forest : tune and test with out-of-bag (OOB) error and data spliting

I would like to perform Random Forest (RF) with a few samples (68 observations to be exact) using r-caret package and the "ranger" implementation on continuous data. So my strategy is to ...
user avatar
0 votes
0 answers
18 views

Evaluation for LSTM model

I have created a model for text generation using LSTM. I am having chess sequences learned, reporting only the pieces moved during the moves. So when I move a pawn on my game there will be "p&...
user avatar
0 votes
0 answers
29 views

What is my best option to evaluate a predictive (or proxy) variable?

I have a list of thunder flashes, and I'm trying to find if a meteorological variable (CAPE) is a good predictable variable (or proxy) for theses flashes. In my thoughts, I want to evaluate this proxy ...
user avatar
0 votes
1 answer
19 views

What metrics work well in unbalanced assemblies?

I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
user avatar
2 votes
1 answer
51 views

Model performance when ground truth is not available

I am building an LSTM Autoencoder (unsupervised model) to detect anomalies in a time series dataset. The input is telemetry data from routers and I want to detect anomalies in the throughout of router....
user avatar
2 votes
1 answer
32 views

Adj. $R^2$ with tree ensembles

Consider tree ensemble methods such as XGBoost, Lightgbm and/or Catboost. Is the adj. $R^2$ a valid metric for tree ensembles? I'm curious because these methods handle factor variables differently. E....
user avatar
  • 218
0 votes
0 answers
11 views

metrics used in search verification, search engines

It's a rather open ended question I think. I'm working on a search engine, and I'd like to quantify my algorithm's performance. I'm aware of search ranking algorithms like pagerank, etc. but the ...
user avatar
  • 231
1 vote
0 answers
18 views

Meaningfully compare target vs observed TPR & FPR

Suppose I have a binary classifier $f$ which acts on an input $x$. Given a threshold $t$, the predicted binary output is defined as: $$ \widehat{y} = \begin{cases} 1, & f(x) \geq t \\ 0, &...
user avatar
  • 224
1 vote
0 answers
22 views

Can I use the AUR under the ROC on unbalanced test data?

I have split my data into training and test data, built several prediction models and now I want to evaluate the models using the test data set.The data is very unbalanced so I balanced the training ...
user avatar
  • 11
0 votes
0 answers
8 views

Scale-independent error metric for data with many zeros

I've been working on a time series forecasting model. I can't use a scale-dependent error measurement. And my target outputs also occasionally have zeros, meaning I can't use MAPE either. What is the ...
user avatar
0 votes
0 answers
19 views

Weighting the R-squared as a measure of goodness-of-fit in Linear Regression [duplicate]

I have two observed time series $x_i$ and $y_i$ and I want to test if $x_i$ is a good predictor of of $y_i$. So I run a simple linear regression Y ~ X and use $R^2$ as a measure of goodness of fit. ...
user avatar
  • 1
5 votes
1 answer
100 views

After training a binary classifier, are TPR & FPR independent of a test set?

Assume I trained a binary classifier $f$ and I was able to extract an optimal decision value $t$ such that the binary output $\widehat{y}$ of the classifier is: $$ \widehat{y} = \begin{cases} 1, &...
user avatar
  • 224
0 votes
0 answers
13 views

Difference between balanced_accuracy score and macro_averaged recall

I understand balanced_accuracy_score metrics metric is recommended as against accuracy_score in imbalanced learning. But one thing I find strange is this measure is always equal to the macro-averaged <...
user avatar
  • 231
0 votes
0 answers
27 views

Accuracy for unfairness detection

I am dealing with unfairness and I am trying to find out some metrics to detect unfairness presence in my dataset. I am starting from the very bases: accuracy. My main question is Do you think I can ...
user avatar
0 votes
0 answers
17 views

confused about sharpness of probabilistic forecasting

In reference [1], it mentioned that "The more concentrated the predictive distributions are, the sharper the forecasts, and the sharper the better, subject to calibration." [1] Gneiting, ...
user avatar
  • 31
0 votes
0 answers
5 views

Why does sklearn list weighted precision and micro precision separately if they are the same thing?

This post explains that micro precision is the same as weighted precision. (And the logic applies to recall and f-score as well.) So why does sklearn.metrics list micro and weighted as separate ...
user avatar
  • 55
1 vote
1 answer
51 views

Measuring the performance of a clustering algorithm with "true" clusters available

Suppose I have observational units $x_1, \dots, x_n$. Each of these units is in a known cluster $C_1, \dots, C_m$, $m < n$. A ML algorithm generates based on a metric new clusters $C^{*}_1, \dots, ...
user avatar
  • 3,883
4 votes
2 answers
221 views

Is there a preference in the regression performance metric for regression models with the same type of loss minimization?

I applied two regression models (ordinary least square (OLS) and linear absolute regression) to the same dataset, where this dataset is split into train and test sets. Two performance measures are ...
user avatar
  • 1,487
0 votes
2 answers
114 views

Is it possible to evaluate a given model without having access to its fit method?

I have a data set with one real-valued feature and a real-valued target. Someone has used this data set to fit a model (a regression). I get a results of this fit, which is a single function mapping ...
user avatar
  • 973
3 votes
2 answers
245 views

A universal measure of the accuracy of linear regression models

I have a dataset that contains both outliers and multicollinearity. I applied three different regression models to that dataset: ordinary least square, absolute linear regression, and Huber regression....
user avatar
  • 1,487
0 votes
0 answers
58 views

Does it make sense that the loss function for traning and evaluaton is different?

Huber loss function is widely used, because it combines the good properties of squared and absolute losses. Therefore, when I apply the penalized regressions, i.e. LASSO, Elastic net and Ridge, to ...
user avatar
2 votes
0 answers
35 views

Predicted Y2 correlates better with Y1 than Predicted Y1

I have a data matrix X of reasonable size (10000, 100), and I am trying to predict targets Y1 and Y2. B1 = (X^T • X)^-1 (X^T • Y1) B2 = (X^T • X)^-1 (X^T • Y2) Pred Y1 = X • B1 Pred Y2 = X • B2 I ...
user avatar
  • 121
1 vote
0 answers
25 views

Bayes Naive classifer on imbalance data [closed]

I read about Bayes Naive classifier will be affected by imbalance data. If so, why still such high roc_auc_score of 0.96? target Proportion Diseased 0.85 Not diseased 0.15 Metric Result Precision ...
user avatar
  • 23
0 votes
1 answer
37 views

How to extract MSEP or RMSEP from lassoCV?

I'm doing lasso and ridge regression in R with the package chemometrics. With ridgeCV it is easy to extract the SEP and MSEP values by ...
user avatar
  • 21
0 votes
0 answers
23 views

same model evidence, different MSE on the test set, which model to report?

I am using Relevance Vector Machines RVM's for regression as from: https://www.jmlr.org/papers/volume1/tipping01a/tipping01a.pdf?ref=https://githubhelp.com. My basis functions are the simple ...
user avatar
  • 101
1 vote
1 answer
38 views

Why will the estimates of prediction error typically be biased upward with Cross-Validation?

Why the estimates of prediction error will typically be biased upward with Cross-Validation? Is it like with decisions tree? Using a stopping criterion will increase a little the bias but will ...
user avatar
0 votes
0 answers
18 views

Comparing performance of models fitted across time series with different frequencies

I have a time series with observations over the span of minutes, hours or even days. I'm interested in knowing if is better to use a daily, weekly or monthly version of the original time series, which ...
user avatar
  • 131
0 votes
0 answers
52 views

Poor out of time performance

I am working on a behavioral model which predicts the probability of default (PD) during the next 12 months for an existing customer with an outstanding loan. My dataset consists of monthly snapshots ...
user avatar
  • 11
1 vote
1 answer
28 views

Generalization of model performance (AUC) and tuning of a catboost classifier

I was wondering if it is good practice to overfit on the training data while tuning a catboost classifier for a binary outcome. Wouldn't it be better to reguralize until validation error equals ...
user avatar
0 votes
0 answers
7 views

How to interpret the lift_curve with a vertical tail?

I know how to interpret lift_curve most of the time. Like what's said in http://www2.cs.uregina.ca/~dbd/cs831/notes/lift_chart/lift_chart.html, Lift is a measure of the effectiveness of a predictive ...
user avatar
  • 289
4 votes
1 answer
120 views

Why custom evaluation/scoring metric is causing overfitting in (cross) validation?

I am using machine learning to approach a balanced binary classification task. Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate ...
user avatar
0 votes
0 answers
22 views

How to evaluate quality of VAEs generated samples

I have a set of generated samples from a latent distribution (say 100 images) from a learned VAE. For GANs, the Inception score metric (which helps assess image quality and image diversity). Any idea ...
user avatar
  • 1
0 votes
0 answers
8 views

Applying ROC to evaluate time-series signal event picking model performance

I am trying to evaluate the performance of an event picking model that attempts to find the onset of a signal in a noisy time series. Data contains the true signal time (ground truth) and the ...
user avatar
0 votes
1 answer
20 views

Ridge regression coefficients show model importance but the model evaulation not

I have performed two ridge logistic regressions in R to check which of the two models perform better. From the first look of the coefficients, it looks like model1 ...
user avatar
  • 332
1 vote
1 answer
27 views

Learning-Agnostic Evaluation of SVM Models

I am at a point where I want to evaluate existing SVM models. For this task I assume I have: SVM model (to make it easier let's say it's a scikit-learn one) Training Dataset that was used to learn (1)...
user avatar
  • 11
2 votes
0 answers
36 views

Final predictive model: using all available data?

Objective: Build a screening tool to identify people at risk of X. Approach: Using data from contexts A and B, we explored logistic regression models to predict X. We did forward & backward ...
user avatar
0 votes
0 answers
23 views

Model Types and Evaluation Metrics for Loan Origination

Our team builds models used to originate loans, and currently, we have a pair of models: Probability of default (PD), which is a classification model, which we evaluate using ROC AUC Loss given ...
user avatar
0 votes
0 answers
32 views

Best Eval Metric for Credit Scoring: ROC AUC/PR AUC/F1?

My team is developing a credit scoring model for a situation in which... The positive class accounts for 10% of the training data FNs (predicting no default for actual default) costs us ~\$10-15K FPs ...
user avatar
1 vote
1 answer
72 views

Model Evaluation: R2 Score is very bad when the range of values is low but Mean absolute error is good

I have a dataset where the values of the dependent variable are small, i.e. range is 1.7 to 2.2. I fitted a model and made predictions on y_test, the predictions ...
user avatar
5 votes
2 answers
463 views

"Dumb" log-loss for a binary classifier

I am trying to understand how I can best compare a classifier that I have trained and tuned against a "dumb" classifier, particularly in the context of binary classification with imbalanced ...
user avatar
0 votes
0 answers
8 views

How to evaluate a GAM that predicts temperature over a certain extent with actual measured data?

I have a GAM that predicts the temperature on a relative scale (0 = coldest, 1= hottest value) of a raster extent. Secondly I have measured data from the same spatial extent that I can transform to ...
user avatar
  • 1
0 votes
0 answers
104 views

What is the equation to calculate the log likelihood of a null model in logistic regression?

I would like to calculate the log likelihood of the null model for a logistic regression manually. Ultimately, this is to calculate McFadden's pseudo-R2, and, yes, I could have software generate the ...
user avatar
5 votes
2 answers
262 views

Are consistently negative Efron's pseudo-r2 in logistic regression possible?

I am conducting logistic regression and looking to calculate pseudo-R2 values alongside AIC and BIC for model evaluation. I selected Efron's pseudo-R2 because of its simple calculation and the ...
user avatar
1 vote
0 answers
29 views

Is there a single regression quality metric for the median and the 95% percentile?

I want to evaluate the quality of prediction of two values the median and 95% percentile of a distribution. Is there a standard way to do this? I have thought about using "Mean Mean Average Error&...
user avatar

1
2 3 4 5
19