# Fisher classification method for normally distributed classes

As a follow-up to my previous question, I want to use Fisher's classification method (I mean, projection method) to project the data given the fact that the two classes are normally distributed. How to approximate the distributions? How to project the data when it's already in a one dimensional space? I remember Fisher's idea (two classes) was to project the data from D-dimensional input space to one dimensional. I don't know why I can't work with real data examples when I know how the actual method works. Please someone guide me through this.

The classes are:

$C_1=\{3, 3.5, 4, 4.5, 5, 5.5, 7\} \cup \{15, 16, 17\}$

$C_2=\{0,0.5,1,2\}\cup\{8,9,10,11,12,13\}\cup\{20,25,30\}$

And the data will be on the interval $[0,40]$ like in the previous question.

I assume you mean Fisher's discriminant analysis or LDA. These are methods for reducing dimensionality in a manner that would be useful for linear separation.

If your data is already in one dimension and not separable, you would need to combine the LDA with another method in order to use it. This would work as follows:

1. Raise your 1D data into a higher dimension using some transformation.
2. Reduce the dimensionality of your data using LDA. Note that now if you reduce to 1D it could be a different dimension then the one you started with.
3. Evaluate linear separability.

I would advise you to look into using a kernel method, such as Kernel LDA which is very similar to doing the above in one step, with the added value that you usually don't have to explicitly represent your points in the higher dimensionality (representation could be a problem at very high dimensions). Also in general kernel methods are very useful and a good thing to know.

• Thank you for your answer. I am taught that for a small data like the one I mentioned in my question where you can see all the data and could find some relationships between them it's not recommended to use kernel methods. Could you please suggest a transformation to a higher dimensional space where the data will be linearly separable? – Gigili Nov 26 '12 at 10:23
• @Gigili a polynomial transformation should do the trick (this would be similar to using a polynomial kernel). To see why, plot x versus x^2. You see that this polynomial of degree 2 already makes part of the subsets which were not linearly separable in 1D into linearly separable in 2D. If you move to a higher degree polynomial you will get full separability. – Bitwise Nov 26 '12 at 16:17

I suggest you consider a Gaussian mixture model like normalmixEM or "nonparametric" (I use that term loosely) mixture model like multmixEM with compCDF, all from the mixtools R package.

A very dumb example of visualizing your data this way:

require(mixtools)
c2 <- list(mix1=c(0,.5,1,2),
mix2=c(8,9,10,11,12,13),
mix3=c(20,25,30))
c2mus <- lapply(c2,mean
c2sigs <- lapply(c2,sd)
c2mix <- normalmixEM(unlist(c2),
lambda=0.5,
mu=unlist(c2mus),
sigma=unlist(c2sigs))
plot(c2mix,density=T)


This is likely too dumb an example:

1. you probably do not want to make strong assumptions about the distro like this, so consider npEM;

2. you probably want to "cluster" each marginal new observation into c1 or c2, in which case build a mixture of regressions using regmixEM and regmixEM.loc

There are great examples of how to accomplish these two methods in the vignette (and this requires some custom fiddling to get a meaningful solution)

HTH,

• That was the right answer, Gaussian mixture model to make the two classes linearly separable. Thank you. – Gigili Dec 1 '12 at 20:11