Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

0
votes
0answers
11 views

What type of accuracy should I reported in research paper?

I have read some research papers on the classification task of deep learning, and now I am doing my own. After investigating some research paper which also provided the source code for reproducing ...
0
votes
0answers
4 views

validating model on small binary dataset with balanced outcome

I have a model which is built on a dataset (N=288), and I want to validate it on another extremely small dataset (N=13) and it is only partly similar to the first one. As they really aren't similar I ...
1
vote
1answer
21 views

Discrimination (pseudo) $R^2$ vs. C-index

In the context of binary logistic regression. Both pseudo $R^2$ and C-index measures the discrimination of the model. But why do you need both ? can you gain something from one but not from the other?
0
votes
0answers
14 views

Variant of validation with singleton test sets

Is the following approach to model validation somehow reasonable? And is there a name for that approach? We have 110 data points, iid assumption holds and we want to compare two predictive models M1 ...
0
votes
1answer
31 views

Model validation in R - Gamma GLMM

I'm trying to model a response variable y with respect to a nested variable x in R. First of all, I fitted a linear mixed model (LMM) as it follows: ...
2
votes
1answer
23 views

Constant validation loss and increasing validation accuracy

I am training a fully convolutional network. The loss is decreasing whilst the validation loss stays mostly where it is. There is some variance in the validation loss. I thought it might overfits, ...
0
votes
1answer
35 views

Training error less than validation error, but higher than test error?

I have a time series regression prediction problem. So I divided the dataset into 3 parts: training (first 70% of the time series data) validation (from 70% to 85% of the time series data) test set (...
2
votes
0answers
15 views

Split dataset before or after preprocessing in training, validation, test

Suppose you have a super large table $T$ with data, which is pretty messy, and it's columns aren't useful features. So you clean it, remove/impute missing values, define new columns, which are more ...
1
vote
1answer
26 views

Different common meanings of training set, validation set, test set [closed]

In this previous question of mine it was pointed out in an insightful comment by ReneBT that the usage of the 3 terms training data set, validation set and test set is not uniform across the cross ...
1
vote
0answers
26 views

Plotting Residuals vs Predicted Values

In textbooks, residual plots are described as have predicted (fitted) values on the x-axis, with the y-axis being the difference between the predicted and observed values. However, I'm having trouble ...
0
votes
3answers
82 views

Machine learning without test and validation data

All mainstream machine learning approaches I've seen depend on a test and usually a validation dataset to measure model accuracy during and after training. This seems like it uses up quite a lot of ...
1
vote
0answers
7 views

factor of safety

I need to validate a specification on a part. The requirement is to withstand a minimum 1 kg of force (i.e. not fail at less than 1 kg). I was told that if I measure, on 6 specimens, a force of 2.5 ...
0
votes
0answers
7 views

Training and validation loss: consistency and interpretation

I have following training & validation loss for my LSTM network. I was wondering what I could deduce from this data. The validation loss seems to start where the training loss ends, is this a ...
0
votes
1answer
17 views

Is backpropagation is used in validation data set? [closed]

Hello guys I am very confused as I am building a deep learning image classifier from raw python code ,so my question is that:-is backpropagation used in validation set to get the model more accuracy ...
0
votes
0answers
20 views

Options when sample doesn't fit any distribution and doesn't pass normality screening?

I have a sample of 30 data points, that I am unable to find a distribution fit for. The goal of my analysis is to assess process capability and get an accurate Ppk. This is one output of a process I ...
3
votes
2answers
73 views

name for histogram of nominal p-values under the null

To evaluate a statistical test or means of generating frequentist confidence intervals, it makes sense to repeatedly simulate data for which the null is true and then compute the nominal p-value, and ...
0
votes
0answers
16 views

Selecting SVM parameters if training data is oversampled/undersampled

I am working on classification for highly imbalanced data. Let's say I have a strategy to oversample/undersample the training data. I plan to use an SVM classifier to perform the classification. Now, ...
2
votes
1answer
76 views

Early stopping on validation set

There exist cases where one can "overfit" on the validation set. Although it is easier to overfit on the training set, the distributions of the validation and test set may not match, in which case ...
0
votes
1answer
25 views

Is it valid to use ROC calculated during test/validation to interpret results of final production model?

I've trained a binary classification model which outputs a "probability" between (0,1). During testing and validation, I use the ROC to measure the performance of the model. Also, I use the ROC to ...
0
votes
0answers
26 views

Is it useful to compute R Squared for regression trees? [duplicate]

I have a regression tree and want to validate the peformance. The first measure I have is the mse to find which model is the best. After that I want to check if the model peforms better then an ...
5
votes
1answer
88 views

Comparing ways to create a composite score

Objective: I have biomarkers $X_1,\ldots,X_p$ (all in continuous scale) and a binary dependent variable $Y$. Because $p$ is large (there are many biomarkers), I want to make a composite score ...
0
votes
1answer
40 views

Interplay between early stopping and cross validation

I am a little bit confused by early stopping and in particular by how it can be inserted inside a CV framework. As far as I understand, I can fix the optimal number of epochs (for NN, or number of ...
0
votes
0answers
25 views

How to convince people that developed predictive model based on Gradient Boosting Machine (GBM) has enough accuracy?

First of all, I'm not a data scientist. I'm an engineer that wants to use machine learning to do a binary classification based on a data that is extracted from computational modeling. I have four ...
0
votes
1answer
20 views

fitting after training and validation

There are a lot written in StackExchange about train-validation-test split of data set. I am confuse with the following. Assume, I trained model using train set. Then I choose model using validation ...
0
votes
1answer
15 views

Statistical test for cross validation with a low mean

I'm working on comparing 2 algorithms with an experimental protocol that produce 100 folds for each one. As a result, I found that my algorithm got (49.29 $\pm$ 1.69) and the baseline got (50.40 $\pm$...
0
votes
0answers
35 views

Model Validation Through Bootstrapping Optimism; p >>> n

On this forum, I've read quite a few posts on the use of bootstrapping a statistic known as optimism when evaluating various models for their out of sample, predictive performance. I personally have ...
3
votes
1answer
38 views

What is the trade off between having a larger validation set versus a smaller one?

Suppose I am comparing several models, e,g, $\{ax\}$, $\{ax+b\}$ and $\{ax^2 + bx + c\}$, $\{ax^3\}$ on data set $\mathcal{D} = \{x_i,y_i\}_{i = 1}^N$ I partition $\mathcal{D}$ into training set ($N-...
1
vote
1answer
19 views

How to validate classification model with ordinal information

I have a Naive Bayes model that predicts 3 classes. As you increase each class it means that the condition is more severe. 0 means no condition, 1 is concern and 2 is that they have the condition. I ...
1
vote
1answer
32 views

Help me interpret my VGG16 fine-tuning results

I have a binary classification problem where I'm trying to classify whether a given cell is cancerous or not. For this I decided to play around with VGG16 pre-trained model and simply remove the last ...
0
votes
0answers
14 views

Data leakage in multilevel validation

I participate in competition that have historical data. I break it down according to this scheme. ...
1
vote
0answers
37 views

Are training-loss optimised embeddings useless? (help resolve a disagreement)

The aim We are training a feed forward neural network as a regressor, with the aim of using the activations of the final layer as a type of embedding vector to represent the input examples. The ...
0
votes
1answer
25 views

Multiple cross-validation and multiple train-test splits

Suppose we have only four observations in a dataset. Let's called them a,b,c and d. If we perform a cross-validation in a k-fold, with k=2, we would get the following : We get two groups of data, (a,...
0
votes
0answers
24 views

Why the score of validation curve is a constant at the beginning and finally?

The sample size is 105 (69 samples about label 0 and 36 samples about label 1) and it contains 10 features. After scaling feature to [0,1] by MinMaxScaler(), I use svc ('rbf'kernel) and 5-fold cross ...
0
votes
0answers
17 views

Effect of Training size in Deep Neural Networks

How can I test for the effect of training size on a Deep Neural Network? Say I have a dataset with 100.000 samples, should I split this in training/validation/test (e.g., 80.000, 10.000, 10.000) and ...
1
vote
0answers
50 views

High loss (low accuracy) on validation set but not on external test set

I'm training a neural network using 70% of my data as training set, 20% as external test set and 10% for validation using Keras. When I evaluate the trained model the performance on the validation set ...
0
votes
0answers
18 views

Regularized linear regression with class imbalance

I am trying to build a Linear Regression model using a not so big dataset. I'm more comfortable doing classification and I am not really an expert in regression. In classification, I was used to ...
0
votes
0answers
47 views

Classification accuracy in holdout similar to CV if set is randomly sampled, completely wrong otherwise

I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process. While experimenting with some baseline models ...
0
votes
1answer
26 views

How can we compute the difference between two silhouette scores for the same dataset?

Given a dataset X on which I applied k-means and I computed the Silhouette Index score. I consider this score as the truth. I applied again k-means on X and I computed the Silhouette Index score. My ...
1
vote
0answers
35 views

Aggregating ROC AUC values of several Logistic Regression Models

I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
3
votes
2answers
59 views

Do I need a validation set if I am doing 10-fold cross validation?

I am looking at a dataset with ~120 observations and I am investigating it using two sets of explanatory variables, one has about 12 features, the other about 8. This is for a regression analysis. ...
0
votes
1answer
70 views

Training loss decreasing, Validation loss steady, where to stop?

In the following training scenario (Orange: training loss, Blue: validation loss), what epoch is the best time for stopping the training? Validation is almost steady as we continue training, but ...
3
votes
1answer
46 views

Does retraining a model on all available data necessarily yield a better model?

A (simplified) typical workflow in machine learning might be: Train $m$ models on a training set. Validate the $m$ models on a validation set to yield the best model with parameters $\theta$. Retrain ...
0
votes
0answers
35 views

Computing and estimating the EER on an entire dataset

I have reproduced "Generalized End-To-End Loss For Speaker Verification". It describes a method to create a deep learning model that can derive an embedding (a vector of 256 float values) that ...
0
votes
0answers
14 views

Dropdown in Validation Loss in the first epochs

I've built a classical backpropagation ANN using Keras for a regression problem, which has two hidden layers with a low amount of neurons (max. 8 per layer). The amount of samples for training and ...
1
vote
1answer
36 views

Should overfitting or underfitting be concerned during hyperparameters tuning

I have built an ANN model using Keras. The problem I'm solving is a regression problem and now I'm trying to tune the hyperparameters. I've found better approaches than using a Grid Search - Hyperopt /...
1
vote
0answers
25 views

80-20 better than full dataset for LightGBM

Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations. I have two modes, 1) Production and 2) Testing. The first one just trains a model with ...
1
vote
0answers
28 views

What is the difference between the validation loss on a regression task and the mean squared error?

The validation loss on regression task using mean squared error loss function is different from the mean squared error value directly calculated on the validation set. What is the difference between ...
0
votes
1answer
20 views

What are reasonable decisions to make when performing logistic regression along with validation?

I'm not really a statistician but, in the words of Scarlet O'Hara in Gone with the Wind, "have always depended on the kindness of strangers.” I have a data matrix corresponding to 20 trials with 15 ...
0
votes
0answers
18 views

Model evaluation - High sensibility and specificity but low MCC

I trained a Random Forest classification model to predict bioactivity for different protein targets. Both my training and test sets were highly imbalanced with ~99% of the majority class. Now that I'...
1
vote
1answer
34 views

How to check if covariates in multiple regression is explaining the same?

I am a master's student doing my thesis at the moment and have come to the point of determining my empirical setup. I would like to get some guidance, in terms of what I am thinking is proper.. I ...