2
$\begingroup$

I have a set of dataset recorded from subjects as they perform some particular cognitive task. The data consists of 16 channels and a number of sample points per channel and I want to classify this data according to the cognitive task being performed (everything is labelled).

The issue is that I do not have a large amount of data (approximately 60 trials per session, 30 for each cognitive task) and I have 2 sessions. I am trying to train a Linear Discriminant Analysis (LDA) classifier to classify this data. The classifier is later to be used in real-time to give some form of output every number of samples.

I made use of a 5-fold cross-validation to measure the generalization error of my classifier. The problem is that when I run this 5-fold cross validation a number of times, the results I get are not constant at all. Instead there's a significant variation in the overall accuracy (for example, first 5-fold cross validation may yield an average accuracy of 80%, 2nd yields an accuracy of 65%, 3rd yields an average of 72% etc...). Is this normal? If not, what could be the causes?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

You may have discovered the fact that LDA models can suffer from instability: slight changes in the training set lead to very different models.

There's a whole body of literature on the variance of resampling methods for validation. This may give you a good start.

See also here for a related discussion. (I'm sure there is a literature list somewhere around here, but I cannot find it right now)

Anyways,

Improve this answer
$\endgroup$
2
  • $\begingroup$ would using some form of regularization scheme for the LDA diminish this problem? $\endgroup$
    – Lunat1c
    Commented Mar 7, 2013 at 9:55
  • $\begingroup$ @user2036690: possibly, yes. Still I recommend that you check first whether instability is really the problem, and after regularization check that things actually improve. If the problem is instability and your regularization is successful, you should also see an increase in hit rate. $\endgroup$
    – cbeleites
    Commented Mar 7, 2013 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.