I'm trying to learn the Bayesian network structure from a very large data set, and the R package I used for learning can only handle a very small portion of the data set (~10%) at one time due to the computational limitations.

The conventional k-fold cross-validation strategy uses k-1 subsets for training and 1 subset for testing.

I want to know if I can use only one random subset for training and another random subset for testing? Is there any better solution?


migrated from stackoverflow.com Aug 23 '12 at 12:56

This question came from our site for professional and enthusiast programmers.

  • 2
    $\begingroup$ In many cases that will lead to poor result. Except you have huge amount of data, using 10% of data to train will not capture the structure of distribution well. you might want to try some technique like bagging to generate many data set with size (say 10% of your original data set) which your algorithm can handle well, in parallel manners , then blend those results you got. $\endgroup$ – Jing Aug 23 '12 at 9:59

The short answer is yes, you can do that. k-fold cross-validation is typically used when sample data is sufficiently limited. From your description, unless your computer is rather resource-limited, it appears that you have a very large sample size. If that is the case (your sample data set is sufficiently large), then you could do a 2-fold cross-validation. This would lead to 2 samples (A and B), where you first train on A and test on B, then vice versa.

As a commenter mentioned, bagging (bootstrap aggregation) is another option, although an ensemble classifier may not be applicable/desirable to your particular problem. Another option would be to randomly draw samples of ~ 10% of the data for training, then another 10% for testing and repeat that process multiple times and assess variability of your results.

What may complicate the analysis is that you stated you are attempting to learn network structure. If the network structure (edges between nodes) is already defined, assessing variability of network weights is fairly straightforward but quantifying variability in network structure (which nodes are connected and in what direction) is a more complicated process.

  • $\begingroup$ The sample size is very large, ~10,000,000 with 20 variables, I think ~10% is the maximum sample size the package can handle. Just like you said, the network structure is unknown, and the learning procedure is rather complicated. I have divided the whole network into multiple modules to reduce the number of learning parameters, nevertheless, I want to make sure the subsets used for learning are representative and the learned structure is robust. $\endgroup$ – Andy Lu Aug 23 '12 at 14:23
  • 1
    $\begingroup$ I think your situation is what most would consider to be a "good problem". A commonly used rule of thumb in machine learning is that you should have a minimum of 10 samples per dimension of your input data and preferably over 100 per dimension. Using that rule, you'd like to have a sample size of over 2,000 observations. Using only 10% of your available data still gives 1,000,000 samples. Considering that your sample size for 10% is about 500 times larger than the minimum desired, you may even want to drop down to 5% and run more iterations of your algorithm for cross-validation. $\endgroup$ – bogatron Aug 23 '12 at 17:15
  • $\begingroup$ @bogatron, do you have a source for the 10/100 samples per dimension claim? I could really use it if you do. $\endgroup$ – Max Candocia Jul 10 '14 at 15:51
  • $\begingroup$ I don't know why I wrote "10" - I think that was a typo. The rule of thumb I recall is a minimum of 30 per dimension/variable but preferably over 100 (per dimension). But, again, it is a "rule of thumb" and not universally applicable. I don't have a specific source but if you do a web search for "rule of thumb sample size", there are many relevant results. You might start here for a list of recommended sample sizes (with references). $\endgroup$ – bogatron Jul 10 '14 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.