Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
0
votes
0answers
29 views

When performing validation, should you also tune the number of hidden units and hidden layers?

I understand tuning the learning rate, momentum, batch size, etc. and finding the best set of parameters using the validation set. However, I don't understand when people say that you should also ...
1
vote
0answers
260 views

What method should I use to cluster small data set?

I would like to cluster a small data sets [23 genes, 50 samples], but I don't know what method I should use... could you give me any recommendation? I have applied hierarchical clustering (Wards ...
1
vote
1answer
226 views

Perceptual Loss Layers Selection

I understand that in order to improve your generative model performance it is quite useful to compare your output and the target in the feature space, as stated in the paper Perceptual Losses for Real-...
0
votes
0answers
52 views

Machine Learning: Why do I have this pattern of train and validation accuracy?

I am trying to understand what would generate this pattern of accuracy in train and validation dataset (second and third plot below). I am training a network to recognize 6 types of faces (they are ...
1
vote
0answers
12 views

Validation of Models with Different Scale Data [duplicate]

I created a model for two different datasets that have different scales. When checking which one performed better, I am struggling with figuring out the best methodology. My top choices right now are <...
3
votes
1answer
111 views

Stability Index for Fuzzy c Means Clustering in R

I am looking for a stability measure/index for Fuzzy C Means Clustering in R. Can anyone direct me to a package or a code in R? I am not looking for internal indices, only stability measures.
0
votes
0answers
46 views

What causes the high OOB-error for randomForest() in R?

I'm trying to perform a random forest in R on a dataset with 16364 observations (after undersampling), using the function randomForest(). But my results look really weird: What could have caused this?...
1
vote
0answers
73 views

Linear regression vs survival models in R

I have a question about survival analysis with a gaussian distribution of the response variable. I have interval censored data, that look something like this (just an example): ...
1
vote
0answers
44 views

Forecasting Validation

question on validating forecast models. The traditional approach (https://otexts.com/fpp2/forecasting-on-training-and-test-sets.html) is to hold-out n number of samples from the end of your time ...
2
votes
2answers
304 views

Do I need a validation set?

Do I need a validation set if I am using cross validation and grid search for parameter tuning? Similar to this question - what I understood is that it helps to prevent overfitting but it's not ...
1
vote
1answer
553 views

A meaning/significance of validation loss in a Generative Adversarial Neural Network? [closed]

On most of the tutorials on GANs that I came across the only monitored quantity is training loss. 1) Are there any general conclusions that could be derived from comparing training and validation ...
1
vote
1answer
324 views

Validation and Test accuracy at random performance, whereas Train accuracy very high

I am trying to build a classifier in TensorFlow2.1 for CIFAR10 using ResNet50 pre-trained over imagenet from keras.application and then stacking a small FNN on top of it: ...
1
vote
0answers
113 views

Test Time Augmentation on Validation set?

In the traditional usage of data augmentations, we augment only the train set examples, in order to keep the distribution of the validation and test set equal. In the TTA method, we apply ...
1
vote
0answers
1k views

eval_set in xgboost and validation data

In my understanding, if I am picking from a set of models, each with a different set of hyper-parameters, the proper way to approach it is like this: First, split the data to ...
1
vote
0answers
38 views

Can you have a concordance statistic (discrimination) when predicting a continuous outcome?

I'm writing a proposal for a prediction model predicting BP (continuous outcome, predicting trend over time). For assessing model performance, I'm seeing discrimination and calibration as the most ...
1
vote
0answers
12 views

What is the difference between validation and verification in statistical models?

We assume errors in our models are additive and uncorrelated, this is a big hypothesis but we'll assume that it is true. This means that observations have an inherent error $\epsilon_{o}$. ...
0
votes
0answers
15 views

How K-Fold Cross-Validation will help in avoiding consumption of training data set?

One way of hyperparameter tuning is to try different values on validation test, suppose we divide the data into training(60%), validation(20%) and test(20%) set. Here disadvantage is we are eating up ...
1
vote
2answers
201 views

Is resampling to obtain a test set ok?

I have a question about cross-validation and hyperparameter optimization. Namely about how to test the final model performance in an unbiased way. Now, I know I have to train the model, optimize the ...
1
vote
1answer
188 views

Validation loss decreasing faster than training loss

I have two different scenarios that I ran across and I can't seem to wrap my head around what caused them. In this scenario, my validation loss, in orange, initially fell faster than my training loss,...
0
votes
1answer
18 views

Validation with only one dataset

I am performing multiple linear regression and multivariate analyses on a dataset (~600 samples). I would ideally use a second, independent dataset to validate my findings, but there is no suitable ...
1
vote
0answers
210 views

Model Validation Using Residual Plot?

I am trying to validate a given multiple linear regression model on a new dataset (a dataset that has not been "seen" by this model). Is it okay to discard this model on account of it showing a ...
6
votes
1answer
3k views

Keras: What is the meaning of batch_size for validation?

If I understand correctly that batch size is the number of samples used in the training of a NN before the gradient gets updated, then why do we need a specified batch_size for the validation sample? ...
0
votes
1answer
21 views

Is this a valid factor analysis approach for validating a multi-domain survey?

I am developing a new instrument consisting of items grouped into defined domains (constructs), such as personality traits. Each construct is measured through a set of 4-6 items (questions). To ...
1
vote
1answer
54 views

When working with unbalanced data, do we train final model on full data set?

Let's say we have an unbalanced data set. We randomly sample an amount from our larger class so that we have a balanced data set. After tuning parameters/hyperparameters and determining which ...
1
vote
1answer
22 views

categorical distribution in validation set

I have a dataset that contains 6877 samples. This is a multiclass multilabel classification which means that we have 9 classes and every sample can belong to one or more of these classes. The total ...
1
vote
1answer
1k views

In sample splitting for time series data, do we randomly select data?

I'm having a hard to conceptually understanding how to do this. I would like to do my own sample splitting (not the method built into a package). Let's say you have 80 days of weather data. You ...
0
votes
1answer
756 views

How to make validation curve more smooth?

I have this validation curve where in for each epoch I plot my validation score and training score. I use the Adam optimizer and don't attain better smoothness when I lower my learning rate, only when ...
0
votes
0answers
16 views

Model selection: testing accuracy higher than cross-validation

I trained convnets with different architetures and validated using 5-fold cross-validation. After this, I trained the final version of the convnets using the full data set (training + validation ...
0
votes
2answers
62 views

Validation criterias in simple linear regression in r

I plotted a simple linear regression in ggplot both for the entire data and training set and test set. But, the problem is that it just provides a plot, nothing more like RMSE, R2, etc. How can I ...
2
votes
0answers
84 views

Any methods for sanity checks on refreshed machine learning models?

I am wondering if there are any best practices for validating ML models that are trained on new data. Apart from the validation metrics used on test data, are there any other recommended approaches ...
0
votes
0answers
31 views

Quantifying difference in model performance?

Probably a very basic question that's been asked before, but ... I have a set of candidate models (mixed-effects linear regressions) I'm fitting to 1/0 (presence/absence) data in R using ...
1
vote
0answers
40 views

Top Decile Lift in Repeated 10-fold Cross Validation

Scope: Compute Top decile lift in the test sets within a repeated(x10) 10-fold cross validation using R. Should I compute the Lift in each fold, and then average all the lifts derived? Then, as ...
0
votes
0answers
67 views

Predicted probabilities very close to 0 and 1 in GLM model

I've added new attributes to the binary GLM model. AUC climbed to 98%, logistic loss decreased to 0.45. Training set has ~50 cases. I can see that predicted probabilities are extremely close to 0 and ...
3
votes
2answers
245 views

K-fold validation for logistic regression in R with small sample size

I used SPSS to develop a logistic regression model from 274 cases. The final model uses 6 independent variables to predict a binary dependent variable with an event rate of 18%. The model is purely ...
0
votes
0answers
230 views

Validation accuracy jumps badly between 0 and 1

Issue: My validation accuracy jumps between 0% and 100%. This seems fairly implausible to me because the predictions are between two classes only and I am validating on full valSet (41802 records) ...
2
votes
2answers
349 views

(Linear regression) Can I train and validate at the same time using the following approach?

In a lot of material I found online, training and validation seems to be an iterative process For example, the regularized regression problem $E = \|Xw - t\|_2^2 + \lambda \|w\|^2_2$ $X$ is data ...
2
votes
2answers
168 views

using the test population as an eval_set when doing hyperparameter optimization

I'm looking at this guide for hyperparameters optimization of boosting regressors using hyperopt. I noticed that for each trial, it uses the following code for the ...
1
vote
0answers
25 views

Evaluating model that is in production without access to new, labeled data

I have a deep learning image classification model that is in production. I am trying to get a sense of model drift by calculating distribution metrics (e.g. mean predicted probability per label) over ...
1
vote
1answer
157 views

About 10-fold cross validation train/test split

So, I want to do 10 fold CV. After I googled it, all of the websites I've found told me that to do the split, take 1 fold as test and the rest as train. But my professor told me another way. She told ...
0
votes
0answers
85 views

In feature selection, what the size of the data set is considered as too small? Is this an appropriate use of machine learning?

I am in a non-computer science field, and machine learning is being blatantly misused in my field. I recently got a journal paper to review, where the researchers used machine learning to develop a ...
1
vote
1answer
26 views

how to interpret the case when the cross validation accuracy is more than the model accuracy

I've trained a ANN model which resulted in 94.62%, but when I do a 5 fold cross validation the mean accuracy is 94.75%. Also 4 out of 5 cross validated models accuracy is more than 94.62%. How to ...
0
votes
1answer
81 views

How to validate Generalized Least Square model for longitudinal response

I have a dataset with body weights before and in the follow-up visits after surgery, for a group of patients with obesity. Our goal is to fit a model to predict weight loss throughout the follow-up. ...
1
vote
1answer
25 views

Should I use a validation set when not cross validating?

I have 3 kinds of models (Lineair regression, random forest regression and the gradient boosted forest regression). Normally I would apply CV to all 3 of them and use a validation set for that and use ...
0
votes
1answer
36 views

Validation loss is decreasing, accuracy is decreasing too

So, I have the following charts from my experience.Can any one explain why accuracy is decreasing while the loss in train and validation is decreasing? The point is that i can't early stop too in the ...
0
votes
1answer
53 views

is it okay to include the prediction data from k-fold cross validation for machine learning training?

Currently, I started to study machine learning to write an academic paper. Lets say I have 1000 data, and I split to 70:30 for training:testing. While training the machine learning (assume binary ...
2
votes
1answer
84 views

How to split training data when learning DNN for unknown test data?

I'm designing a CNN model for a data mining competition in which we are provided with N sample of training data. We do not know the test size, but presumably it is from the same distribution as ...
0
votes
1answer
140 views

Why does my OLS model produce high error on test set?

To create the feel of using my model to predict unknown data, I split the dataset I have into training set and test set. The $R^2$ value of my OLS regression model in the training set is decent (89.16%...
18
votes
4answers
3k views

Is leave-one-out cross validation (LOOCV) known to systematically overestimate error?

Let's assume that we want to build a regression model that needs to predict the temperature in a build. We start from a very simple model in which we assume that the temperature only depends on ...
0
votes
0answers
98 views

How do I know I divide dataset into training set, validation set and test set in a correct/appropriate proportion?

Chapter 3 of Tom M. Mitchell. Machine Learning (free) gives this advice: One common heuristic is to withhold one-third of the available examples for the validation set, using the other two-thirds ...
0
votes
0answers
26 views

Unsure how to continuously train a churn model after my model has gone live

I'm having trouble describing this properly so I'll provide as many details as possible. First, here are a few details on the model that I am building: I have built a classification model that ...

1 2 3
4
5
16