Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
0
votes
1answer
157 views

Why does my OLS model produce high error on test set?

To create the feel of using my model to predict unknown data, I split the dataset I have into training set and test set. The $R^2$ value of my OLS regression model in the training set is decent (89.16%...
18
votes
4answers
3k views

Is leave-one-out cross validation (LOOCV) known to systematically overestimate error?

Let's assume that we want to build a regression model that needs to predict the temperature in a build. We start from a very simple model in which we assume that the temperature only depends on ...
0
votes
0answers
99 views

How do I know I divide dataset into training set, validation set and test set in a correct/appropriate proportion?

Chapter 3 of Tom M. Mitchell. Machine Learning (free) gives this advice: One common heuristic is to withhold one-third of the available examples for the validation set, using the other two-thirds ...
0
votes
0answers
31 views

Unsure how to continuously train a churn model after my model has gone live

I'm having trouble describing this properly so I'll provide as many details as possible. First, here are a few details on the model that I am building: I have built a classification model that ...
1
vote
1answer
108 views

Common practice for validating classifiers in medical statistics

We are writing a paper, where we suggest a new classifier. We have a data base with about 1400 medical cases. Would it be sufficient just to divide the dataset into training 70% and test 30%? Or ...
0
votes
1answer
74 views

Estimate of an AR model

I have this part of a project which states this: Once you choose the best three models for each series (according to AIC, the PACF and ACF, and "from general to specific), the next step is to estimate ...
0
votes
1answer
127 views

Loss and Accuracy computation

I have a doubt about the computation of the loss and accuracy for the training / validation / test. I split my training set into batches, and I'm training my network with them for N epochs. My idea ...
17
votes
5answers
9k views

Can increasing the amount of training data make overfitting worse?

Suppose I train a neural network on dataset A and evaluate on dataset B (that has a different feature distribution than dataset A). If I increase the amount of data in dataset A by a factor of 10, is ...
0
votes
1answer
20 views

Sample Size to Validate if Report is Pulling Correct Information

I am having a report written on very complex EHR data where the data needed comes from different formats, from different locations. I need to choose a sample from about 20,000 accounts to manually ...
1
vote
2answers
2k views

What is the most appropriate way to validate prediction models with clustered data?

I am attempting to develop and validate a multivariable classification model using data from 10 clinical trials. I would like guidance on the most appropriate way to validate (internally and ...
-1
votes
1answer
21 views

Rather use a sample of a population or analyze a whole population but smaller?

I am validating an automated surveillance program. It is about infection incidence on a hospital ward. I could easily check if the small list of infections produced by the automated surveillance ...
1
vote
1answer
557 views

Detecting overfitting on multi-class classification model

I have seen this question asked in one flavor or another, but I'm looking for clarity on a more specific piece. I have two text classification models: Model A: train score=88%, test score=76% Model ...
9
votes
2answers
227 views

When should I use Validation rather than Cross Validation

I am aware that CV was born as a way to validate models when there is a lack of training data, but my understanding is that it is generally better to cross validate rather than just use one validation ...
0
votes
0answers
172 views

Comparing AUC, logloss and accuracy scores between models

I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem: ...
1
vote
1answer
45 views

Area under the curve in risk prediction model

The area under the curve for my data set is $0.63$. However, when I divided my data randomly into two parts, development (67%) and validation (33%), the value of the area under the curve became $0.58$...
0
votes
1answer
88 views

Machine Learning - Training/Validation Sets

I have a very general question that I can't seem to get a straight answer on. Machine Learning - I understand how it works - you have your dataset for which you want to answer either a prediction or ...
1
vote
0answers
33 views

Can test results be considered reliable if it is not possible to have true out of sample validation data set?

We are working with an anonymized dataset (similar to commercially available consumer data) to create a binary classification based model. We have data for entire US population that is commercially ...
0
votes
0answers
19 views

How can underfitting and overfitting Classifiers of the same Type perform equally worse?

An rbf support vector classifier is instantiated with different values for C and gamma. Then learning curves are plotted for each of the classifiers. Some classifiers consistently achieve very high ...
2
votes
2answers
168 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
0
votes
1answer
130 views

Validate Z-score Formula

I need need help validating a formula for a z-score. I have a book that I enjoy but in my re-read of the materials, I have found that the author has made some fundamental errors in their writing. So, ...
3
votes
0answers
69 views

How do you evaluate the prediction accuracy of linear mixed models?

How does one evaluate prediction accuracy with uncertainty for linear mixed models? Let's say I do bootstrapping and do train/test each time, and want to generate confidence intervals for some ...
0
votes
1answer
48 views

How to check if a Machine Learning model is applicable for newly input data?

Suppose we have a good Machine Learning model, with good cross-validation and test score. How can we estimate whether a newly input data instance belongs to the domain of data where model predictions ...
3
votes
0answers
55 views

A goodness of fit test for two discrete distributions with unequal variance?

I have a computer simulation in which a virtual agent end up in different areas of a layout based on several factors. There are 18 conditions, so the data (you can find the csv file for a toy dataset ...
0
votes
0answers
42 views

Getting the distribution of a model accuracy metric using the Posterior of the Model Parameters

Say I compute the posterior via MCMC of a classification model's hyper-parameters $\theta$ given my observed data: $\pi(\theta|D)$. Would it be at all useful to take a look at, for example, the ...
4
votes
1answer
528 views

How did these researchers determine the confidence interval of the AUROC using resampling but without retraining the model?

In this Nature article backed by Google, the investigators develop then externally validate a deep learning model for predicting lung cancer using CT scans. In their internal validation results, we ...
3
votes
1answer
1k views

Early stopping together with hyperparameter tuning in neural networks

Similar to this question (hyperparameter tuning in neural networks), I have a neural network with a similar list of parameters as the link above: Learning rate: $[0.001, 0.01, 0.1]$ $L_1$ penalty: $[...
0
votes
0answers
17 views

inferential approach for estimating error rate on classified population

I am looking mainly for ideas and approaches which I could not find by just Googling. I created a classification model to predict about 175 unique classes from text features. I trained the model on ...
1
vote
0answers
1k views

Interpreting calibration plots

This is some kind of really specific/multidisciplinar question, so maybe this is not the appropiate place to expose this, as it's quite code-heavy. If that's the case, please inform me! I have fitted ...
2
votes
1answer
2k views

Why my validation accuracy and AUC are higher than my training accuracy and AUC?

I have a binary classification problem and I use LightGBM classifier to build my model based on 5 features. I divided my dataset (94 observations) into two parts: Training dataset: 60 observations ...
1
vote
0answers
551 views

Understanding stratified K fold cross validation results (for LSTM binary classification model) [duplicate]

I am performing a Binary Classification task with LSTMs. Data_size (205, 100, 4) - Out of 205 samples 110 belongs to class 0 &...
0
votes
1answer
55 views

How can i account for the error in a model when using that model to predict data?

I have a regression model with $R^2$ value. I am trying to use this model on a different data set to see how well the model predicts this data. So far I have used the model with the 'new' data to ...
1
vote
1answer
23 views

When to validate your data? [closed]

I have simulated a scenario under ideal conditions. This simulation was done using the Friis equation to determine the power received by a RFID tag for a particular frequency, distance, power of the ...
2
votes
1answer
120 views

Validating uncertainty quantification

Regression performance is often evaluated by means of cross-validation. However, classical cross-validation only regards the mean of the identified parameters. How can one quantify the quality of the ...
0
votes
1answer
71 views

How to correctly find the best hyperparameter combination when training a neural network?

I am not sure whether this is the right place to ask this question, so feel free to redirect me if not. What I'm doing is bench-marking a model (MobileNet v2 100 224) in terms of performance - size ...
2
votes
0answers
141 views

OOB error prediction in RF if case weights are used

I have a dataset for which grossing-up factors are given. I am using these factors as case weights for a random forest (R package ranger). Until now I was using the OOB prediction error for tuning, ...
0
votes
1answer
529 views

How to evaluate Regression Model Accuracy using Caret

I am training a regression model using caret in R but I am having a hard time understanding how to validate the model. I am using postResample and it is outputting and RMSE and an R squared. I am ...
0
votes
1answer
46 views

Labels in Out-of-time validation

I have a binary classification problem and I use out-of-time validation to validate my models. My question is regarding the label. There is a lag in identifying the correct label. Simplified example: ...
3
votes
0answers
37 views

When and how to re-evaluate deployed models

There are a lot questions about how to train a model (family) or how to tune hyperparmeter. But there are surprisingly few question about how to monitor or evaluate a model already in production (...
1
vote
1answer
37 views

Statistical validaty of heuristic labelling

I have an unlabelled dataset with over 10K observations. For label assignment, I used some heuristic that I devised from first principles. The heuristics where in the form of "If the attribute ...
0
votes
1answer
92 views

Determine by AUC and RMSE if Logistic Regression or Decision Tree is a better model?

I am using Linear Regression and Decision Tree to predict whether an e-mail is spam or no spam. I have built both models and got different values regarding AUC and RMSE. Can I determine by AUC and ...
2
votes
1answer
437 views

Validation accuracy/loss goes up and down linearly with every consecutive epoch

I'm training a CNN in keras with tensorflow backend with the following model architecture for a binary classification problem. I've divided approximately 41k images into training, validation and test ...
1
vote
1answer
626 views

Word2vec Skip-Gram - Overfitting

i am currently training a skip-gram model on my own dataset. After each run i compare the cosine-similarity between all the vectors and get the following diagramm: So my model creates each run nearly ...
1
vote
1answer
336 views

Validation Accuracy Increases But Training Accuracy Doesn't

The training accuracy of my model is not improving though validation accuracy improves steadily. This is weird abnormal behaviour and I just can't figure out what's wrong. Here are some graphs to help ...
4
votes
1answer
2k views

Somers' D for model validation

Somers' D as defined for example in "The predictive accuracy of credit ratings: Measurement and statistical inference" by Walter Orth is defined for the case when Y is predicted by X as $$ D_{XY} = \...
0
votes
1answer
454 views

Data augmentation on entire dataset before splitting

If I apply rotation of 5 different angles and randomly cropp 10 different images from each rotated image and than divided into training testing and validation. Will it be totally incorrect evaluation ...
0
votes
0answers
103 views

Time series data validation error is significantly lower than training error

I have a time series dataset that covers daily observations (closing price) for several stocks, and I would like to build models to forecast the closing prices for the future 7 days using their ...
-1
votes
2answers
532 views

Train/Test split for imbalanced regression problem

I have a dataset with ~100K samples and continuous target variable which has 95% of zero values. Since there are high-dimensional categorical features and missing values in my data, I plan to use ...
0
votes
0answers
460 views

What is wrong with tuning parameters on training set as opposed to validation set?

When creating a machine learning model it is suggested to split your data into train, validation, and test sets. Here is my understanding of what they are for. Train: Use this to train the different ...
0
votes
0answers
299 views

What type of accuracy should I reported in research paper?

I have read some research papers on the classification task of deep learning, and now I am doing my own. After investigating some research paper which also provided the source code for reproducing ...
3
votes
1answer
519 views

Discrimination (pseudo) $R^2$ vs. C-index

In the context of binary logistic regression. Both pseudo $R^2$ and C-index measures the discrimination of the model. But why do you need both ? can you gain something from one but not from the other?

1
3 4
5
6 7
16