Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

0
votes
1answer
15 views

Differences in calibration plots for machine learning models

I'm using machine learning methods in R for descriptive regression modelling of a small dataset. I have fit random forest (randomForest), unbiased random forest (cforest) and boosted regression trees (...
1
vote
0answers
25 views

Validating a model built on multiple regressions

I have a program that models suspended sediment concentrations (SSC) using turbidity as a predictor and lab derived sediment concentrations as the response. The relationship between the two can ...
0
votes
1answer
27 views

Is splitting data randomly into train, validation and test sets a bad idea?

In Splitting into train, dev and test sets it is recommended that It is important to choose the dev and test sets from the same distribution and it must be taken randomly from all the data. I have ...
0
votes
0answers
18 views

Difference of network between testing and training on the same dataset (No training and testing)

I was training and dense net model on emotion recognition on the sewa dataset. Therefore, at the end I have 2 outputs. One for arousal and the other for valence (These dimensions for emotions). So I ...
0
votes
0answers
8 views

Can overfit happen in spite of validation and what to do with it?

Let's consider a standard situation where we need to find a predictive model. We train all the available model using a training data set. We validate all the trained model using a validation data ...
0
votes
0answers
8 views

Meaing for MOAC in Spherical Payoff

I want to implement this metric Spherical Payoff mentioned in both articles and Netica software to validate my bayesian network (through a test dataset), here are the formula that I got from my ...
0
votes
1answer
35 views

Hold out validation. Exactly what is left out?

I'm studying validation and I've seen multiple examples of hold out validation. Some will hold the tail of the data, while others will leave out $n$ random points. I assume it has to do with whether ...
0
votes
0answers
14 views

How to validate the results of bayesian causal network?

There are many ways of validating predicting the results: MSE, MAE, AIC, CV, etc.. But I do not hear any validation way of causality. If the true networks not available, how to make sure the results ...
0
votes
0answers
30 views

What is a good way to test a bidirectional recurrent neural network?

I wrote an implementation of a bidirectional RNN. How can I verify that this implementation is correct by using a small dataset? Is there any simple test that compares performance of bidirectional ...
0
votes
0answers
10 views

Is it appropriate to perform outlier treatment on test sample data set? I am building logistic regression model

I am building logistic regression model. Is it ok to perform outlier treatment on significant variables after building the model and if yes, do we need to perform outlier treatment on test sample data ...
0
votes
1answer
16 views

Should the validation set have the same ratio in the categories as the whole data?

I'm currently working on a classification problem. The variable Y in 70% of cases is 0 and in 30% of cases is 1. Does my validation set have to have this same proportion? I ask because after using ...
0
votes
1answer
22 views

H2o k-fold validation [closed]

I need to get some clarification on how H2o creates a training model from the k-fold validations. Below is my understanding, please correct where I am wrong: If I set nfolds = 5, then H2o will split ...
0
votes
0answers
14 views

how to validate a questionnaire that will be used for small population?

I created a questionnaire (likert scale) to measure "an implementation of school principles". the questionnaire will be given to teachers and staff. the population is just 20 persons. should I do ...
0
votes
1answer
18 views

Question about Validation Set for hyperparameter tuning

Okay, I'm still a bit confused as to this Training/Validation/Test Set split. I might be wrong here, but from what I understand, the model is first applied to the Training set, to "learn" from it and ...
2
votes
1answer
60 views

Bias corrected calibration curve (regression modelling strategies)

I have a question regarding calibration plot for a binary logistic regression model (calibrate) in the rms(regression modelling strategies) package. The Bias-corrected curve (see below) shows if the ...
0
votes
0answers
23 views

How to split data for stateless LSTM in Keras? [duplicate]

I have been thinking about the way of splitting the data into training, validation and test sets for the stateless LSTM. For me, the intuitive way is to arrange the original data into the 3D form (...
0
votes
0answers
14 views

Residuals of a model using the training set vs the testing set vs the full set

I have a Gamma GLM with a log link function to predict face amounts of insurance. This model was created using the training set, which is 75% of the full data, randomly sampled. Now that I have this ...
0
votes
0answers
15 views

Imbalanced class SVM prediction results using different validation data

I am trying to fit my data to a classifier using SVM. My data has 2 classes, the positive class which occurs with a probability of 0.002 and the negative class which is the dominant one. Suppose that ...
8
votes
4answers
176 views

Is it mandatory to subset your data to validate a model?

I'm having a hard time getting on the same page as my supervisor when it comes to validating my model. I have analyzed the residues (observed against the fitted values) and I used this as an argument ...
0
votes
0answers
9 views

Validation of logit model

I have performed econometric analysis with binary logit model using national representative survey (weighted) data for a given country. I then received a suggestion that no tests or robustness ...
0
votes
1answer
23 views

The representation of F1-score on the Precision-Recall Curve

Is there a way to project the F1-score on the precision-recall curve for a such binary classifier? Is there a relationship between the area under the precision-recall curve and F1-score? ...
0
votes
2answers
54 views

Linear predictor from coefficients of Cox PH model

I need to calculate the linear predictor of a Cox PH model by hand. I can get continuous and binary variables to match the output of predict.coxph (specifying 'lp') but I can't seem to figure out how ...
1
vote
1answer
34 views

Is it valid using testing data for model selection when validation data cannot do the job?

In Hastie's Elements of statistical learning, it says: "The training set is used to fit the models; the validation set is used to estimate prediction error for model selection; the test set is used ...
0
votes
0answers
31 views

Test Set Probabilities and Accuracy [duplicate]

Say we've got a logistic regression model $M$ used as a classifier in a binary case. Now we take a test set $\tau=\{(x_1,y_1),...,(x_n,y_n)\}$, each test sample is assigned with $\hat{\pi}_i=P(y_i=1|...
1
vote
0answers
37 views

Overfitting decreases over time

During my training I have got the following results after 10th epoch: training accuracy amounts to 97% and is stable; training loss is decreasing from 0.2530 and is decreasing 0.02-0.05 per epoch, ...
1
vote
1answer
29 views

sklearn.train_test_split truncates my y_train [closed]

I have a standardized dataset which has floats with this (for example, e+01) tail at the end. I know this is a multiplicator to save such a small number without ...
2
votes
3answers
134 views

Does multicollinearity cause type I errors?

Citing Wikipedia on multicollinearity, One of the features of multicollinearity is that the standard errors of the affected coefficients tend to be large. In that case, the test of the hypothesis ...
0
votes
0answers
57 views

Training RMSE higher than Validation RMSE in H2O

I am using the H2O-DeepLearning Model for a Regression Problem. What i observe is that Training RMSE is higher than Validation RMSE. I am using the model with default parameter which is two hidden ...
0
votes
0answers
29 views

Calculation of RMSEC (Calibration Error), Which data should be used?

I got a bit confused about RMSEC. I got how RMSECV and RMSEP are calculated and their meaning, though. I am working based on 2 papers which summarize their results with RMSEC, RMSECV1, RMSECV2 and ...
3
votes
2answers
55 views

External validation of clustering requires labels, but why cluster at all if you have labels?

There are two types of validation in clustering, using: Internal indexes: Used to measure the goodness of a clustering structure without respect to external information (e.g., sum of squared errors)...
0
votes
0answers
14 views

What should be the size of testset?

If the obtaining of data is expensive, I want to use as less data as possible. I am not looking for an answer like 40% of all data, or 10000 is enough. I think it relates to how precisely I want to ...
0
votes
0answers
17 views

Can a worm plot be used with a count output variable?

I have a Gamlss model with a Sichel output variable and i am not sure if it is ok to use worm plot for validation. currently the worm plot looks like this:
1
vote
2answers
76 views

Questions about performing k-fold cross validation for the first time

I am learning how to perform K-fold CV and I have a few questions about the method. This is an excerpt from the website: https://towardsdatascience.com/cross-validation-in-machine-learning-...
2
votes
0answers
66 views

Optimism bootstrap with non-linear models

I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper ...
0
votes
2answers
34 views

Can I create a test dataset with known errors to validate accuracy assessment? [closed]

I developed a procedure to measure the geometric accuracy of 3D building models based on the similarity to a 3D point cloud. Therefore I created mainly two quality criteria. The result of my automatic ...
-1
votes
1answer
17 views

Which internal validation index I can use on similarity matrix

I'd like to evaluate my clustering results using only internal indices. For example Silhouette index (S) validates the clustering performance based on the pairwise difference of between- and within-...
0
votes
1answer
26 views

Is there a systematic reason why a model trained on a subset of data does better out-of-sample than the same model trained on the full dataset?

I trained a linear regression model using 3000 data points. (OLS regression, no regularization.) Then I trained another model with the same predictors (about 25), but with a subset ($n=700$) of the ...
1
vote
1answer
55 views

in-sample data vs out-of-sample data

I know that a train-validation-test splits the data into: a training dataset - obviously my "in-sample" data a validation dataset a test data set - obviously my "out-of-sample" data My question is: ...
0
votes
0answers
61 views

Should I augment validation set?

I am doing image recognition with neural nets and applying image augmentation to extend train set and lower overfitting. Should I apply augmentation to validation set too? Currently I don't and ...
0
votes
1answer
45 views

What is the best way to test and validate a multivariate regression using OLS?

I am implementing a multivariate regression from scratch using Ordinary Least Squares to get the weights. I noticed that this method does not have any hyperparameters to tweak, so I am not sure what I ...
0
votes
0answers
8 views

How should I split my observation map for calibration and validation of a rule-based model?

I have a map of observed vegetation classes (more than 20 classes) that was digitized from a aerial foto. And I developed a rule-based vegetation model to predict these classes. The model calibration ...
1
vote
0answers
7 views

Is it possible to “tune” a trained model in one population so it can be used for a different population (i.e., by swapping variables)?

Say, I have trained a model to classify patients into cardiovascular disease (CVD) and non-CVD. The model building process is as follows: There is a gold standard to compare the model with. The ...
0
votes
0answers
11 views

Is pseudo-r2 enough to validate a CART?

I run some Classification and Regression Trees (CARTs) and computed the pseudo-$R^2$ from McFadden. Is that enough to validate the trees or do I need some other test to be sure there is no overfitting?...
2
votes
0answers
12 views

What should I do when individual variables don't validate in a test set?

I built a model for purposes of prediction that validates out well overall on a test set. However, when looking at individual variables, there appears to be a couple (out of 100) that show no pattern. ...
0
votes
1answer
71 views

When we should NOT use k-fold cross validation to assess the predictor?

Does anybody know in which cases --of learning and predicting--, it is better to use "validation test" or something else, instead of "k-fold cross validation" to assess the performance of the ...
1
vote
2answers
103 views

Techniques to avoid overfitting

I have heard of several techniques to avoid overfitting: Validation curve: which let us choose the set of parameter with the minimum step between validation score and training score. But it seems ...
1
vote
0answers
18 views

Arima vs HoltWinters

I am currently performing some sales forecasting and I am debating between an ARIMA model, using the auto.arima function in R, or a HoltWinters model, using the HoltWinters function in R. What are ...
0
votes
0answers
27 views

Choosing a model

I am working on a sales forecast right now and I have created 4 models but I am unsure which one to use. I have 17 Quarters of data(4 Full years + 1 QTR) and I am only looking to forecast 2 quarters ...
0
votes
0answers
27 views

validation of UPGMA clustering

UPGMA and single-linkage represent two different forms of hierachical clustering techniques. Similar to a regression analysis, a phylogenetic correlation may be determined in the range of 0 (bad model)...
0
votes
0answers
70 views

How to evaluate a regression model?

I have a dataset with 190 samples. I'm training a regression model, using k-fold cross validation. I'm using $R^2$ for measuring the performance of the model. However, sometimes, depending on the ...