I have the following dataset where
binary_peak is a binary response variable and I have (not shown) 9 explanatory variables (also binary).
`binary_peak` H3K18Ac H3K27me3 H3K36me3 1:00 0 0 0 0 2:00 0 0 0 0 3:00 0 0 0 0 4:00 0 0 0 0 5:00 0 0 0 0 --- 1903462: 0 0 1 0 1903463: 0 0 1 0 1903464: 0 0 0 0 1903465: 0 0 0 0 1903466: 0 0 1 0
I am a little bit confused about the cross validation procedure. The way I am currently doing this is fitting a model on all 1.9 million rows.
r1 = glm(formula = binding_peak ~ 1 + H3K18Ac + H3K27me3 + H3K36me3 family = binomial(link = "logit"), data = massive_ds)
After this, I run $K = 10$-fold cross validation on the entire dataset again.
cv.glm(data = massive_ds, glmfit = r1, K = 10)
I believe my approach here is wrong. What I should be doing is splitting the entire dataset into two (or three?) sets:
- Training Set
- Validation Set
- Test Set (??)
Does this mean that when I fit my model and perform K-fold validation, I am ONLY using the training set? I was under the impression that this is exactly what the K-fold cross validation does? That it breaks my entire dataset into groups, uses one group to train a model, and then apply the model on the remaining groups.
Also, how do I then apply this model to the dataset? My goal is to create ROC curves characterizing the model's accuracy, but if I am using the entire thing as a training/validation set (interally), would it be sufficient to just apply the model again on the training set?
Some background: I have data on biologically significant areas of the entire mouse genome. The genome is split into bins of 200 basepairs, and the response variable (binary) indicates whether a bin is of interest or not. Once I get confirmation that I need to, indeed, split my entire dataset I would take
chr 1 - 6 as a training set and use the rest as a validation set.