Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
-1
votes
0answers
8 views

Recoding of the ERQ (Emotion Regulation Questionnaire) items? [closed]

For an assignment, I have to check the reliability and validity of a questionnaire I contextualized for the working population. I decided to contextualize the ERQ (Emotion Regulation Questionnaire). ...
0
votes
0answers
10 views

How to use sample mean to validate the quality of the mean of model predictions?

We have a large set of training data based on ground truth samples from specified locations, but we are most interested in calculating means within a certain area of interest (AOI). We arrive at these ...
0
votes
0answers
7 views

Is an internal validation or test set required if there is an external/independent dataset?

I have two datasets collected independently. Each has about 150 observations. I would like to to develop a support vector machine/classification model on one dataset, and assess its performance on the ...
0
votes
0answers
5 views

Examples of Adversarial Validation Use Case

I am new to machine leaning, and learning basic concepts. I understand that Adversarial Validation is a method used to tell if our train and test data are similar . if we can classify them (ROC > ...
0
votes
0answers
20 views

How to decide model performance on validation data

I have built several neural networks to decide what choice of a hyperparameter is best. I want to use the validation data (not test data) to do this. To determine the performance of each model should ...
0
votes
0answers
7 views

Do I use the same validation data to determine all the hyperparameters for a neural

I have a 10000 piece dataset and so I will split it 70/15/15% for training, validation and testing. For feature selection I intended to use all the data and mRMR (by analysing weights of a trained ...
0
votes
2answers
27 views

How to decide model parameters of a neural network effectively

When choosing neural network parameters say numbers of features, layers and neurons, is the best way to do this by training each of the options several times by cross-validation and then take the ...
1
vote
1answer
17 views

Why would you use a subset of the training set as the validation set in parametric classification?

I was told by my ML lecturer that the validation set, as used in parametric classification, is used to determine how overfitted the model is to the training data, but that it is also a subset of the ...
0
votes
2answers
38 views

Random initialization of weights

I have trained a neural network using a train and a validation dataset.I used the validation dataset for hyperparameter and architecture optimization.I split the dataset in the exact same way each ...
0
votes
1answer
18 views

Validating the percent variance explained of principal components on out-of-sample data

I'm trying to ascertain whether the variance explained by a certain PCA on an out-of-sample dataset is not due to random chance. Suppose I have a dataset X with size n-x-p, and I run a PCA and obtain ...
4
votes
1answer
84 views

Validation of a linear mixed effect Model

I am setting up models of the diversity data (Shannon Index) of bees and hoverflies to find out which in which studied seed mixture (site) the diversity was higher. In addition, I would like to know ...
3
votes
4answers
481 views

How many data points for test set in a time series

I have a monthly sales data set from 2018 January onwards. I would like to know from expert what is the optimum train test split and minimum train test split. Also to mention that my data includes ...
0
votes
1answer
22 views

Two different types of K-Fold Cross-Validation. Are both ways right ? Advantages or disavantages of each?

For brevity let's say that k=10. 1º Scenario: I'll divide the training dataset in 10 parts of equal size, train my models in 9 groups joined together and measure some metrics on the remaining one, ...
1
vote
1answer
21 views

How is validation of neural nets implemented?

I know what are training, validation and testing stages. But, I want to know how validation is implemented. Let's consider the following data : data_train, data_val, data_test as respectively training,...
0
votes
0answers
10 views

Lift Charts and Significant Predictors

Suppose I have two models, Model A and Model B. Suppose each of them have the same set of predictors, the only difference is ...
0
votes
1answer
11 views

Should models built using under-sampled data be evaluated against the population

I have a dataset of 11 mil. rows with a 1:10 ratio between minority and majority classes. To train a model, I have selected all the minority class members and 1/3 of the majority class. The ratio is ...
1
vote
0answers
25 views

Validation of Clustering with labels

I am currently trying to perform clustering on 8 different datasets where I have 40-100 "labeled" data points per data set, representing which data points belong to the same cluster. I ...
1
vote
1answer
40 views

Should one use the usual splitting (Learning/Validation/Test) when using cross-validation?

Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets: learning set: used to build your model; validation set: used to ...
0
votes
0answers
22 views

Data Split Strategy

In case of very less data set for training and testing, can I remove validation set and keep only train and test set for training a DL network for segmentation? If yes, then how to get the best model ...
3
votes
1answer
35 views

Why do we choose the hyperparameters that gives the lowest validation error? Do we assume that it also gives the lowest generalization error?

The usual way of selecting hyperparameters is to tune it on the validation set and select the hyperparameters that gives the lowest validation error (Lets assume the validation sample is large so we ...
1
vote
1answer
27 views

Reconciling one-way ANOVA in MATLAB with Results from R

I'm used to work in MATLAB but want to be able to use R as well. Currently, I'm trying to run a one-way ANOVA in R using the following example from MATLAB using the hogg dataset showed here: https://...
0
votes
0answers
6 views

Cumulative sum of Mean Decrease Accuracy above 100%

Why is it that sometimes the cumulative sum of the Mean Decrease Accuracy of several input variables after a Random Forest Classification lays above 100 %? Example. How can a model lose more than 100% ...
1
vote
0answers
14 views

Have I correctly specified my GAMM model?

I am new to GAMM models and I would like to ask you whether I designed the model correctly. I aim to evaluate the relationship of a blood protein (independent variable) with the disease scoring system ...
1
vote
0answers
17 views

Cant understand training and validation accuracy [closed]

Im trying to train a model to classify some data in 6 categories. I get the following graphs using tensorboard Im using Adam optimizer with learning rate 0.001 and I dont understand Why the training ...
0
votes
0answers
11 views

Why is there a obvious tilt in Random Forest validation results?

I'm using RF model by sk-learn in Python. I've already trained the model, and the density scatter plot for validation on testing sets looks like below: A severe problem is that the slope of the ...
0
votes
0answers
5 views

How can I evaluate smoothed offline predictions for later real-time use without future bias?

I have a system that makes predictions in real time. Whenever the system encounters a positive label, the process must be stopped. This, however, means that noisy predictions, such as the one shown at ...
0
votes
0answers
10 views

LSTM batch error in validation vs error in whole validation

I am training an LSTM for different series (passing them all at once). While training I print the MAE of each training batch and validation batch. At the end of the epoch, I print the "epoch ...
0
votes
1answer
22 views

hyperparameter search with unknown test set distribution

I'm training a 3-class neural network classifier (conv layers and softmax at the end, nothing special). Let's say, in the test set I will have N1 examples of the 1st class, N2 examples of the 2nd ...
0
votes
0answers
11 views

How can I perform ensemble subset selection based on the OOB error?

As we all know, selecting the optimal random forest based on the out-of-bag (OOB) error is an efficient way to determine the best model without an additional validation set. However, it appears that ...
0
votes
0answers
20 views

How to deal with data augmentation for training neural networks?

I'm trying to apply some data augmentation techniques for training my neural network model. I know that I need to avoid including synthetic data generated from test data in the training data. Also, I'...
0
votes
1answer
33 views

How validate machine learning models with imbalanced datasets?

I'm trying to train a neural network model. Let us suppose that I have a dataset with 4 classes: Class 1 - 500 samples Class 2 - 2000 samples Class 3 - 15000 samples Class 4 - 60000 samples In my ...
2
votes
1answer
17 views

Assessing the performance of a model that has a policy that makes it worse

Assuming that a churn model will be used to prevent customer churn using policies that encourage customers with higher churn score (higher likelihood of churn probability) to stay loyal to a company. ...
1
vote
1answer
43 views

What is the difference between a validation and a development set?

For most of the ML problems we have train,test and validation sets in a dataset as discussed in this thread. I have a dataset where I have train, development and evaluation sets. train and evaluation ...
1
vote
2answers
43 views

Cox prediction models: Statistical inference versus cohort-split (derivation->test)

First I want to clarify that I understand that all prediction models needs external validation, and this applies to both machine learning models and conventional regression models. My question is ...
0
votes
0answers
18 views

Is it necessary to evaluate model performance for logistic regression inferencing (vs prediction)?

I'm newer to the world of statistics, forgive me if this is a rudimentary question. I transitioned from point-and-click SPSS to Python, and am realizing that one can use logistic regression for ...
0
votes
0answers
22 views

Model fitting VS model selection: what works best?

Suppose we have two candidate models to predict a variable $y$ given a variable $x$, where $\alpha$ is a model parameter. $$\hat y=M_1(x,\alpha)$$ $$\hat y=M_2(x,\alpha)$$ Conceptually, we could ...
1
vote
0answers
8 views

If you want to train a model on the predictions of another trained model, would you use the same training set for both model?

Imagine I train a model and make predictions on it, denoting the training data by x_train and the predictions by y_pred_train. Then, I want to use those predictions in a second ML model. Would you use ...
2
votes
2answers
179 views

Why validation accuracy is increasing very slowly?

My convolutional network seems to work well in learning the features. However, the accuracy of the validation set is increasing very slowly with respect to the learning rate as also illustrated in the ...
1
vote
0answers
56 views

Data Generating Process Simulation for Entity Resolution

Entity resolution across multiple databases with millions of entities seems to be quite a laborious learning task. Right now, I am combining a variety of methods to come up with confident estimates. ...
0
votes
0answers
15 views

Entering actual standard errors for c-statistics using metaprop (meta package in R)

I am trying to meta-analyze c-statistic using the meta::metaprop function in R. However, I noticed that I can only enter the number of events and the total sample size with the function. When I do ...
0
votes
1answer
37 views

How to aggregate log-likelihood score of many models?

I have a process through which I estimate the parameters of a model in order to make predictions. Through this process, I end up with many models (with different parameter values) that are then used ...
0
votes
1answer
33 views

Train vs Test performance

Hey I am building logistic model. My train data has 4000 observations and my test set has 1000 observations. What suprised me is fact that for train set I get AUC 0.9 but on my test set I get AUC 0.92....
40
votes
3answers
4k views

Why is it that my colleagues and I learned opposite definitions for test and validation sets?

In my master's program I learned that when building a ML model you: train the model on the training set compare the performance of this against the validation set tweak the settings and repeat steps ...
0
votes
0answers
9 views

Validation loss very high [duplicate]

The validation loss is decreasing very slowly and the accuracy after 100 epochs is significantly lower than the train accuracy. What seems to be the problem here and how can I fix it? Do I decrease ...
0
votes
0answers
8 views

PCA and Regression for Modal Split values

I want to predict the modal split values for cities in Germany (percentage bike, car, pedestrian, public transport). I am owning data for the modal split of the biggest 81 cities in Germany. Further, ...
0
votes
2answers
82 views

(How) can model parameters be learnt using MCMC?

I get stuck by part (b) of figure 4 in this paper: Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users. In my understanding, inference algorithms like MCMC are not for training ...
0
votes
0answers
10 views

When using clustering can we use hypothesis testing to find out whether means are different in each cluster?

I have used a clustering method to segment a dataset and I used some methods to determine the number of clusters. However, I wanted to use hypothesis testing to investigate if the clusters have the ...
0
votes
0answers
35 views

Test accuracy much higher than validation and train accuracy

My test accuracy is around 20% greater than validation accuracy and the train accuracy. I have read a few posts and some suggest that there might be something wrong with the split. In the below code I ...
0
votes
0answers
12 views

Logistic model - differences between development and validation

I have some data (eg. Titanic) and I want to use logistic regression to predict probability of survive. I have problem to understand the difference between model development and model validation. ...
0
votes
1answer
79 views

Validation of inverse problem solution based on Bayesian method

Recently, I read a paper about the inverse problem and parameter estimation. The main approach of the paper is based on the Bayesian method. The answer in this method is a posterior probability ...

1
2 3 4 5
16