1
$\begingroup$

I have a signal X that depends on parameters Y (condition) and S (participant). I'm interested in classifying Y based on X, but a) the YxS contingency table of numbers of observations per Y and S classes is unbalanced and b) classifying S based on X yields high accuracy. Thus, I am worried about S-specific information contributing to the accuracy in classifying Y based on X, i.e., I want to make sure that the classification of Y based on X cannot be explained by S-specific information embedded in X.

Technically, this may correspond to a mixed-effects MANOVA problem, but this is a bit too involved for my purposes as the classification needs to be repeated numerous times.

I came up with the following solution: fit a linear model with the formula X = 1 + S, and regress out S-specific information from the data based on the resulting coefficients. (i.e. remove the Bi*Si part from X)

This results in a corrected matrix of data Xc where means across observations are very close to identical across classes of S.

To verify that no S-specific information is present in the data, I attempted to decode S based on the corrected data Xc. I'm doing pairwise classification, so I was expecting to get ~50% accuracy.

However, I'm getting ~2.5% accuracy, well below chance. To clarify, I'm running this classification hundreds of times and consistently get below chance classification with a mean of about ~2.5% and most of the classifications actually get 0% accuracy (versus 50% for chance). Weights are consistent with the notion that no S-specific information could be extracted. For all intents and purposes it looks like indeed S-specific information is removed from the data, so that any remaining classification of Y cannot be attributed to S. However, I'm confused about the below chance classification performance of S in the (near) absence of information about S in the signal.

I'm using a linear SVM (libsvm), and using the '-w1' etc. class weight flags to offset bias when dealing with unbalanced data.

I've looked at the "corrected data" Xc to try and understand the "issue". I'm wondering if this is something akin to "regression towards the mean": all S-specific means are equal in the dataset, so if there is a difference between S classes in the training set then the difference is likely to be reversed in the test set?

Is there any more meaningful way to control for S-specific information and verify that it has been removed?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.