2
$\begingroup$

I'm interested in testing whether relationships from 1 site can be used at another. The plan is to perform kfold crossvalidation (fit model to 80% of data, predict and compare to the other 20%) and repeat this numerous times. However, site A and site B have approximately 20k obs and site C has 20M obs. When I split 80/20 with all obs, do I need to take into account the different number of obs from each site? After the multi site split, I 'm going to split it within each site and then compare model performance when all sites are considered independently vs all sites considered together. Thanks

EDIT I think this is related enough to be an edit rather than a new question. I'm also interested in whether the performance metrics will be affected if the validation sets are different sizes for each site. After proportionally sampling from each site, I might do something like:

skillfun=function(dF,model){
    pred=predict(model,dF)
    data_frame(r2=cor(dF$snowdepth,pred)^2,mae=mae(dF$snowdepth,pred))    
}
valdata %>% group_by(basin) %>%
    do(
        skillfun(.,model)       
        )

valdata contains the points NOT sampled for the fitting process. However valdata will have different number of points for each site again. Does this affect the comparisons of r2 and mean absolute error between basins?

$\endgroup$

1 Answer 1

1
$\begingroup$

When you fit your models for each site independently it doesn't matter, but your estimates will just be more precise in the bigger site. I'd use the same $k$, though, just to be sure. What you should take into account is to make a stratified sampling for your combined model. That is, each of the $k$ folds should have a proportion of each site similar to the proportion of each site to the total.

$\endgroup$
4
  • $\begingroup$ and so does the validation data set need to have the same number of observations when I compare r2 and mean absolute error? $\endgroup$
    – Dominik
    Dec 8, 2015 at 0:59
  • $\begingroup$ Could you clarify please or open another question? $\endgroup$ Dec 8, 2015 at 1:03
  • $\begingroup$ edited the question. I think its related to the initial question well enough. $\endgroup$
    – Dominik
    Dec 8, 2015 at 2:14
  • $\begingroup$ I think it is okay to do that, because you have less info for the other sites in the first place, but keep in mind those results will be less accurate. $\endgroup$ Dec 8, 2015 at 3:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.