6
$\begingroup$

I recently read the paper by Loeppky et al. on Choosing the sample size of a computer experiment: a practical guide and was curious to know if there were rules of thumb about the sample size considerations for Gaussian process (GP) classification? The paper suggests a value of $n=10\times d$ where $d$ is the dimension of the input space, however, this result is for GP regression and not the GP classification case. I imagine classification probably requires more points since it seems to be a harder problem but am not sure if anything has been published on the topic. If it matters, in my application the dimension of the input space is NEVER larger than 1.

$\endgroup$
  • 1
    $\begingroup$ I'm not sure this is really knowable (but I'm not voting to close). If the signal is strong and the model correctly specified, the problem is much easier (can get away with a smaller sample size) than a problem with many irrelevant features and weak signal. $\endgroup$ – Sycorax Apr 6 '16 at 18:59
  • $\begingroup$ @C11H17N2O2SNa I am just asking for a rule of thumb not a hard number on the sample size. $\endgroup$ – RustyStatistician Apr 6 '16 at 19:09
  • $\begingroup$ GPs will give you the uncertainty in your predictions (provided a good prior and kernel selection). That gives you a tool to estimate how many samples you need until you get to the uncertainty level you want. Not aware of a rule of thumb. $\endgroup$ – Bar Oct 13 '17 at 10:38
3
+100
$\begingroup$

Classification can need more points than regression, for a few reasons:

1) If there are only 2 classes, the response variable contains much less information than a continuous variable, which could take many values.

2) With a small number of points, it's especially easy to get complete separation, which makes the maximum likelihood estimate undefined.

With ~10 points, it would be really difficult to tell what's going on, even if your predictor variable is one-dimensional. This graph was generated randomly with 10 independent coin flips. Even though one could easily imagine adding a curve that wiggled up and down to fit the points better, doing so would be a mistake. For this reason, it probably doesn't make sense to even start thinking about nonlinear classifiers before you have a few dozen points. This graph was generated randomly

At the very least, you'd want to heavily constrain the GP model to keep it from overfitting, and/or have a pre-determined kernel based on first principles. With so few points, the model-fitting process is likely to either blow up (if your constraints are too weak) or produce something that just matches your initial assumptions (if your constraints are too strong).

$\endgroup$
0
$\begingroup$

that rule of thumb is wrong. GP is used in high dimensional settings with d >> n without any problems (genomics, mri) lets say 500 000 voxels in one image used to classify 100 subjects into 2 classes. I don't know if there is any theoretical bound, actual number would depend a lot on the nature of the data at hand, so how strong is the signal and what proportion of features are pure noise.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.