3
votes
2answers
116 views

k folds cross validation on a multi-class dataset

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
0
votes
0answers
33 views

Validation of model with relatively sparse data

I am not a statistician, so please forgive my lack of precision with terminology in the following background and question. I have done a fair amount of reading and research about the following, but ...
1
vote
0answers
55 views

Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
1
vote
1answer
140 views

How can I validate a logistic regression model using averaged parameter estimates?

Let me say thanks in advance. I'm working with a set of data that contains reported coyote sightings. I use 2/3 of the data for model calibration along with an equal number of pseudo absences. I ...
2
votes
1answer
582 views

How do I validate my multiple linear regression model?

I have split my data into two parts. I have used my 80% data to build a multiple regression linear model. Now I want to test it using my rest 20% data. What tools on Minitab do I have to make this ...
0
votes
0answers
137 views

Mean squared error — internal and external validation

This is probably a very easy question... but I am not quite sure what the right answer should be.. I have validated 3 beforehand selected models. I have done internal cross validation and I have ...
0
votes
0answers
308 views

How to write a loop in R to select multiple regression model and validate it?

I would like to run a loop in R. I have never done this before, so I would be very grateful for your help ! I have a sample set: 25 objects. I would like to draw 1 object from it and use it as a ...
3
votes
1answer
54 views

Mixing User Data For Cross-Validation

I have data from 12 users, and want to perform cross-validation. Is it necessary that I create my training and test data from different users, or can all of the data be randomized, and then split into ...
0
votes
1answer
128 views

Using components from PCA as index / measurement?

I have used Principal Component Analysis to find main components that desribe the perceptions of the environment. The goal is to use those main components as index / measurement for another study. My ...
2
votes
1answer
82 views

Is a different CV arrangement the same as a validation set?

I have a smallish dataset ~ 1500 rows X 500 columns. I've been using a standard 5 fold CV setup where row 1 = CV set1, row2 = CV set2, ... row 6 = CV set1,etc. I'm at the point where I'm trying to ...
2
votes
2answers
565 views

What are acceptable validation or cross validation error rates?

Is there a commonly acceptable error rate for validation? As in, if the error rate is less than X %, then my machine learning method would be considered "successful". I'm looking for something ...
7
votes
2answers
1k views

What is the procedure for “bootstrap validation” (a.k.a. “resampling cross-validation”)?

"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
0
votes
1answer
241 views

What is a good academic citation for cross-validation?

I performed a holdout cross-validation analysis on a multilevel model fit. The purpose of this was to show that we didn't have a problem with over-fitting, for which it worked just fine. Now we are ...
1
vote
1answer
117 views

Difference between Step validation and Cross validation

I'm trying to model some chemical data from a series of spectra. I've gone through the preprocessing step and selected "autoscale" (I'm using Pirouette). And when it comes to validation I'm a little ...
3
votes
1answer
595 views

How to choose training and test sets

I would like to propose a single model (decision tree), that is very variable, and validate it. I have choosen parameters after I had obtained good quality measures with a cross-validation. I could ...
1
vote
0answers
158 views

Minimal number of samples/conversions for statistical validity

We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country). ...