1
$\begingroup$

I have two datasets which each contain two classes (four classes total). I suspect that both datasets contain the same kinds of classes (i.e. the boundary that distinguishes classes is similar across datasets, and the classes are the same across datasets)

Dataset1: class1=A, class2=B

Dataset2: class1=A, class2=B

As opposed to...

Dataset1: class1=A, class2=B

Dataset2: class1=C, class2=D

I want to test this idea using a nonlinear classifier (e.g. neural network) as my primary tool. The idea I have is to:

1) Train a classifier (via cross-validation) to distinguish class1 & class2 on dataset1. Measure performance/accuracy using test set and all that jazz.

2) Apply pre-trained classifier from step 1 to dataset2 to see how well it can separate the two classes within dataset2

3) Compare classification accuracy between testSet_dataset1 and testSet_dataset2 to assess how similar/different the class discriminating boundary is across datasets.

With a nonlinear classifer such as a neural network, this classifier approach would likely be better than simply comparing distributions using the original feature representations (e.g. w/ KS test) due to the fact that the classifier will hopefully notice nonlinear patterns in each observation (which are likely vital to capturing the underlying stats of each class).

Now for my questions:

1) How should I assess whether or not the accuracy difference in step 3 is SIGNIFICANTLY similar/different?

2) Would such a detected similarity/difference allow me to state that the datasets contain matching classes, or will I only be able to state that the boundary between classes are similar?

3) Would it be more straightforward/preferred to project my data into a lower-dimensional space (perhaps via an autoencoder to extract patterns from the feature space) followed by running a KS test on suspected identical classes across datasets? This would allow me to assess class similarities across datasets as well, while also utilizing more abstract/pattern-esque feature representations.

$\endgroup$
  • $\begingroup$ You could use a kolmogorov-smirnoff test to see if the distributions were statistically different $\endgroup$ – OliverFishCode Mar 13 at 19:57
  • $\begingroup$ If you are only using data set A for training, what exactly are you classifying? This is unclear since all of your training data seem to be from the same class. $\endgroup$ – knrumsey Mar 13 at 20:25
  • $\begingroup$ @knrumsey Apologies for confusion. Will clarify in post. $\endgroup$ – qualiaMachine Mar 13 at 20:57
  • $\begingroup$ Thanks for the clarifications. How many instances of each class do you have? Are they about the same? $\endgroup$ – knrumsey Mar 13 at 21:28
  • $\begingroup$ I do not. Observation count will be different for each class in every case (both within and across datasets). $\endgroup$ – qualiaMachine Mar 13 at 21:31
1
$\begingroup$

Let me suggest a related but slightly different procedure which utilizes resampling. Let's re-frame the problem a little bit.

You essentially have four data sets $A$, $B$, $C$ and $D$ each with $N_A, \cdots N_D$ observations respectively. Let $N=\min(N_A, \cdots N_D)$ and randomly partition each data set into two sets so that the first set contains $n < N$ observations and the second set contains $N-n$ observations. For example $A_1$ and $A_2$ could each contain $n=N/2$ observations.

You want to test the hypothesis $$H_0: \text{decision-boundary}(A,B) = \text{decision-boundary}(C, D)$$ Use your neural-network to learn $\text{decision-boundary}(A, B)$ by training on $A_1$ and $B_1$. Now we use this decision boundary as a classifier for the set $(A_2, B_2)$ and compute the accuracy which we denote $T_{AB}$. Now use this decision boundary a second time as a classifier for the set $(C_2, D_2)$ and compute the accuracy $T_{CD}$. Finally, compute the statistic $$T = T_{AB} - T_{CD}$$ If you null hypothesis is correct, we have $E(T) = 0$.

Suppose you perform this procedure and find that (for example) $T= 0.1$. We need to know if this value is sufficiently far from $0$. We can do this via resampling. That is, we repeat the procedure $M$ times, computing $T^{(m)}$ for each repetition. This procedure, very much like a non-parametric Bootstrap, allows us to construct an approximate sampling distribution for the statistic $T$. Intuitively, if the sampling distribution of $T$ is centered or nearly centered at $0$, then there is little evidence to suggest your hypothesis is false. An approximate empirical two-sided p-value for this procedure is given by $$\frac{2}{M}\times \min\left(\sum_{m=1}^MI(T^{(m)} < 0), \ \sum_{m=1}^MI(T^{(m)} > 0) \right)$$ where $I(\cdot)$ is the indicator function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.