4
$\begingroup$

I am currently using a Bayesian network model with 20 variables and 210 data points, with 15 locations measured at 14 different time points each. There are also some restrictions on what types of connections are allowed.

I have looked at leave-one-out cross-validation methods (and arguably leave-two-out would be computationally feasible with this sample size), and I am currently using that method. However, I want to know which of the following, if any, are appropriate means of going about this:

  1. k-fold cross-validation, leaving out a single location (14 points). My main concern is that if the model were to be applied to locations not sampled, it is quite possible that correlations within a single locations's set of measurements may be unnecessarily influential on a model. Additionally, it's been suggested that k-fold cross-validation produces better results for simpler models such as linear regression. [1] Since my sample size is small, however, I'm not sure how much I can afford to cut out.

  2. Bagging (bootstrap aggregating)[3]. I've mostly seen this method applied to decision trees, but it's used to avoid overfitting.

  3. Jacknifing to determine the properties of the estimators being derived.

Additionally, one interesting property (at least for the datasets used in Zuk et al.2) of the Bayesian Networks is that underfitting decays exponentially fast with sample size, whereas overfitting decays as a power of N. Keeping that in mind, I am trying to decide what methods are most appropriate for my scenario.

Sources used:

1 FAQs.org: What are cross-validation and bootstrapping?

2 Zuk et al. "On the Number of Samples Needed to Learn the Correct Structure of a Bayesian Network".

3 Elidan, Gel. "Bagged Learning Structure of Bayesian Networks".

Edit: In addition to the above methods used, it has been suggested to perform confirmatory factor analysis as an additional method to determine if groups of connections make sense. It doesn't tell you anything about the signs/magnitudes of relationships, though.

$\endgroup$
  • $\begingroup$ I think leave one out is the best you can do. I have a strong aversion to any bootstrapping techniques statisticians employ to get around small sample sizes - it's basically cheating, it has zero logical justification, it's only used because it has nice properties. $\endgroup$ – samthebest Jul 18 '14 at 10:25
  • $\begingroup$ One thing is not clear to me: Are you trying to determine the network structure or just its parameters? If it's the latter case, I'd recommend a full Bayesian inference (treating every parameter as a latent variable) instead of going the Maximum Likelihood/Posteriori way. It handles small data (even zero data) very well because it never overfits, at worst your parameters will have huge uncertainty. If it's the former, I honestly have no idea. $\endgroup$ – Pedro Tabacof Sep 17 '15 at 18:17
  • $\begingroup$ At the time, I was looking at the former (network structure). The sample size was increased a bit with new data, so it was less of an issue, but there were still some stability issues. $\endgroup$ – Max Candocia Sep 17 '15 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.