Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

1
vote
0answers
10 views

High loss (low accuracy) on validation set but not on external test set

I'm training a neural network using 70% of my data as training set, 20% as external test set and 10% for validation using Keras. When I evaluate the trained model the performance on the validation set ...
0
votes
0answers
17 views

Regularized linear regression with class imbalance

I am trying to build a Linear Regression model using a not so big dataset. I'm more comfortable doing classification and I am not really an expert in regression. In classification, I was used to ...
0
votes
0answers
42 views

Classification accuracy in holdout similar to CV if set is randomly sampled, completely wrong otherwise

I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process. While experimenting with some baseline models ...
0
votes
1answer
21 views

How can we compute the difference between two silhouette scores for the same dataset?

Given a dataset X on which I applied k-means and I computed the Silhouette Index score. I consider this score as the truth. I applied again k-means on X and I computed the Silhouette Index score. My ...
0
votes
0answers
19 views

Aggregating ROC AUC values of several Logistic Regression Models

I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
3
votes
2answers
46 views

Do I need a validation set if I am doing 10-fold cross validation?

I am looking at a dataset with ~120 observations and I am investigating it using two sets of explanatory variables, one has about 12 features, the other about 8. This is for a regression analysis. ...
0
votes
1answer
25 views

Training loss decreasing, Validation loss steady, where to stop?

In the following training scenario (Orange: training loss, Blue: validation loss), what epoch is the best time for stopping the training? Validation is almost steady as we continue training, but ...
3
votes
1answer
29 views

Does retraining a model on all available data necessarily yield a better model?

A (simplified) typical workflow in machine learning might be: Train $m$ models on a training set. Validate the $m$ models on a validation set to yield the best model with parameters $\theta$. Retrain ...
0
votes
0answers
22 views

Computing and estimating the EER on an entire dataset

I have reproduced "Generalized End-To-End Loss For Speaker Verification". It describes a method to create a deep learning model that can derive an embedding (a vector of 256 float values) that ...
0
votes
0answers
11 views

Dropdown in Validation Loss in the first epochs

I've built a classical backpropagation ANN using Keras for a regression problem, which has two hidden layers with a low amount of neurons (max. 8 per layer). The amount of samples for training and ...
1
vote
1answer
32 views

Should overfitting or underfitting be concerned during hyperparameters tuning

I have built an ANN model using Keras. The problem I'm solving is a regression problem and now I'm trying to tune the hyperparameters. I've found better approaches than using a Grid Search - Hyperopt /...
1
vote
0answers
18 views

80-20 better than full dataset for LightGBM

Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations. I have two modes, 1) Production and 2) Testing. The first one just trains a model with ...
1
vote
0answers
16 views

What is the difference between the validation loss on a regression task and the mean squared error?

The validation loss on regression task using mean squared error loss function is different from the mean squared error value directly calculated on the validation set. What is the difference between ...
0
votes
1answer
18 views

What are reasonable decisions to make when performing logistic regression along with validation?

I'm not really a statistician but, in the words of Scarlet O'Hara in Gone with the Wind, "have always depended on the kindness of strangers.” I have a data matrix corresponding to 20 trials with 15 ...
0
votes
0answers
15 views

Model evaluation - High sensibility and specificity but low MCC

I trained a Random Forest classification model to predict bioactivity for different protein targets. Both my training and test sets were highly imbalanced with ~99% of the majority class. Now that I'...
0
votes
1answer
33 views

How to check if covariates in multiple regression is explaining the same?

I am a master's student doing my thesis at the moment and have come to the point of determining my empirical setup. I would like to get some guidance, in terms of what I am thinking is proper.. I ...
0
votes
0answers
37 views

ROC curve for multi-variable based prediction in a 3 class classification

I have a data with 10 variables (continuous with log transformed values) that I am using to accurately predict in a 3 class classification. I used RF model to select those 10 variables by first ...
1
vote
0answers
20 views

Error for Validation of Cox model with extern dataset using rms val.surv by Frank Harrell [closed]

i developed a cox model for cancer overall survival with the rms-package. I use a with 765 observations (194 intern, 571 extern). So I divided in two datasets: Training and Validation I get a correct ...
1
vote
0answers
34 views

Subsampling as a method for time series train/validation splits

I have a question concerning train-test splits for time series data: Background I have a dataset of sensor data points for 13 month with datapoints measured every 5 minutes which I downsample to ...
1
vote
0answers
21 views

Selecting a validation set for time-series data

Question1: how much data (how many days) should I select for validation given I have very little data to go on. Question2: given that this is a textbook temporal data set situation here, how do I ...
1
vote
1answer
33 views

Is it possible to calculate F-value for a neural network regression model?

I trained a model using neural network regression and used the F-value equation that is used for calculating F-value for linear regression: F=(SUM(Ypredicted-Ymean)^2/p)/(SUM(Ypredicted-Yobserved)^2/...
0
votes
1answer
20 views

When the validation set is a subset of the training set

I am doing the following but I am not sure if this is right or which behavior should I expect: A union B union C is the full dataset Training set: is A union B datasets Testing set: is C Validation ...
0
votes
0answers
6 views

Shape of the validation loss

I am trying to understand if the validation loss should decrease constantly or can have the shape I am having in this case. I wonder because the validation accuracy does grow constantly as expected. ...
0
votes
1answer
84 views

Python / Keras: SMOTE and validation_split

I try to train a MLP with an imbalanced dataset. I'd like to use SMOTE to balance my classes; as highlighted here (https://beckernick.github.io/oversampling-modeling/), the class rebalancing should ...
1
vote
1answer
36 views

Machine learning model underperformance on unseen data

This is a follow-up question to a question I had previously posted on this forum We conducted an experiment on 100 subjects and obtained a dataset that was used to train a machine learning model that ...
0
votes
0answers
25 views

Goodness of Fit Tests for Logistic Regression [duplicate]

I am trying to build a logistic regression model from a data set with several different features, and ~500 entries. Some of these features are discrete, some are continuous, and some are binary. What ...
1
vote
0answers
18 views

Tests for predictive models with autoregressive neural networks

I'm working with time series predictions with NNAR autoregressive neural network models (p, P, k) and I'm doubtful for the validation of my models. After making the predictions, I'm selecting those ...
0
votes
1answer
20 views

Do I include the validation set in final training?

For optimizing an unsupervised neural network with 1 hidden layer, I use the training set for training and the validation set for optimizing the number of neurons in the hidden layer (for example by ...
0
votes
1answer
39 views

Differences in calibration plots for machine learning models

I'm using machine learning methods in R for descriptive regression modelling of a small dataset. I have fit random forest (randomForest), unbiased random forest (cforest) and boosted regression trees (...
1
vote
0answers
28 views

Validating a model built on multiple regressions

I have a program that models suspended sediment concentrations (SSC) using turbidity as a predictor and lab derived sediment concentrations as the response. The relationship between the two can ...
0
votes
1answer
33 views

Is splitting data randomly into train, validation and test sets a bad idea?

In Splitting into train, dev and test sets it is recommended that It is important to choose the dev and test sets from the same distribution and it must be taken randomly from all the data. I have ...
0
votes
0answers
20 views

Difference of network between testing and training on the same dataset (No training and testing)

I was training and dense net model on emotion recognition on the sewa dataset. Therefore, at the end I have 2 outputs. One for arousal and the other for valence (These dimensions for emotions). So I ...
0
votes
1answer
24 views

Can overfit happen in spite of validation and what to do with it?

Let's consider a standard situation where we need to find a predictive model. We train all the available model using a training data set. We validate all the trained model using a validation data ...
0
votes
0answers
14 views

Meaing for MOAC in Spherical Payoff

I want to implement this metric Spherical Payoff mentioned in both articles and Netica software to validate my bayesian network (through a test dataset), here are the formula that I got from my ...
0
votes
1answer
37 views

Hold out validation. Exactly what is left out?

I'm studying validation and I've seen multiple examples of hold out validation. Some will hold the tail of the data, while others will leave out $n$ random points. I assume it has to do with whether ...
0
votes
1answer
27 views

How to validate the results of bayesian causal network?

There are many ways of validating predicting the results: MSE, MAE, AIC, CV, etc.. But I do not hear any validation way of causality. If the true networks not available, how to make sure the results ...
0
votes
0answers
32 views

What is a good way to test a bidirectional recurrent neural network?

I wrote an implementation of a bidirectional RNN. How can I verify that this implementation is correct by using a small dataset? Is there any simple test that compares performance of bidirectional ...
0
votes
0answers
13 views

Is it appropriate to perform outlier treatment on test sample data set? I am building logistic regression model

I am building logistic regression model. Is it ok to perform outlier treatment on significant variables after building the model and if yes, do we need to perform outlier treatment on test sample data ...
0
votes
1answer
20 views

Should the validation set have the same ratio in the categories as the whole data?

I'm currently working on a classification problem. The variable Y in 70% of cases is 0 and in 30% of cases is 1. Does my validation set have to have this same proportion? I ask because after using ...
0
votes
1answer
37 views

H2o k-fold validation [closed]

I need to get some clarification on how H2o creates a training model from the k-fold validations. Below is my understanding, please correct where I am wrong: If I set nfolds = 5, then H2o will split ...
0
votes
0answers
15 views

how to validate a questionnaire that will be used for small population?

I created a questionnaire (likert scale) to measure "an implementation of school principles". the questionnaire will be given to teachers and staff. the population is just 20 persons. should I do ...
0
votes
1answer
37 views

Question about Validation Set for hyperparameter tuning

Okay, I'm still a bit confused as to this Training/Validation/Test Set split. I might be wrong here, but from what I understand, the model is first applied to the Training set, to "learn" from it and ...
2
votes
1answer
77 views

Bias corrected calibration curve (regression modelling strategies)

I have a question regarding calibration plot for a binary logistic regression model (calibrate) in the rms(regression modelling strategies) package. The Bias-corrected curve (see below) shows if the ...
0
votes
0answers
29 views

How to split data for stateless LSTM in Keras? [duplicate]

I have been thinking about the way of splitting the data into training, validation and test sets for the stateless LSTM. For me, the intuitive way is to arrange the original data into the 3D form (...
0
votes
0answers
18 views

Residuals of a model using the training set vs the testing set vs the full set

I have a Gamma GLM with a log link function to predict face amounts of insurance. This model was created using the training set, which is 75% of the full data, randomly sampled. Now that I have this ...
0
votes
0answers
22 views

Imbalanced class SVM prediction results using different validation data

I am trying to fit my data to a classifier using SVM. My data has 2 classes, the positive class which occurs with a probability of 0.002 and the negative class which is the dominant one. Suppose that ...
8
votes
4answers
205 views

Is it mandatory to subset your data to validate a model?

I'm having a hard time getting on the same page as my supervisor when it comes to validating my model. I have analyzed the residues (observed against the fitted values) and I used this as an argument ...
0
votes
0answers
10 views

Validation of logit model

I have performed econometric analysis with binary logit model using national representative survey (weighted) data for a given country. I then received a suggestion that no tests or robustness ...
0
votes
1answer
75 views

The representation of F1-score on the Precision-Recall Curve

Is there a way to project the F1-score on the precision-recall curve for a such binary classifier? Is there a relationship between the area under the precision-recall curve and F1-score? ...
0
votes
2answers
111 views

Linear predictor from coefficients of Cox PH model

I need to calculate the linear predictor of a Cox PH model by hand. I can get continuous and binary variables to match the output of predict.coxph (specifying 'lp') but I can't seem to figure out how ...