Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

The fisher linear classifier for two classes is a classifier with this discriminant function:

$h(x) = V^{T}X + v_0$

where

$V = \left[ \frac{1}{2}\Sigma_1 + \frac{1}{2}\Sigma_2\right]^{-1}(M_2-M_1)$

and $M_1$, $M_2$ are means and $\Sigma_1$,$\Sigma_2$ are covariances of the classes.

$V$ can be calculated easily but the fisher criterion cannot give us the optimum $v_0$.

Then, how we can find the optimum $v_0$ analytically if we are using fisher classifier?

share|improve this question
1  
$v_0=(M_1+M_2)/2$ seems to be a relatively good candidate (at least by symetry)? –  robin girard Nov 26 '10 at 15:44
    
@robin: although $v_0=(M_1+M_2)/2$ or $v_0=P_1M_1+P_2M_2$ (where $P_1$ and $P_2$ are a priori probabilities of classes) are good heuristic solutions, i need the optimum $v_0$. Is it possible to determine it? –  Isaac Nov 26 '10 at 15:51
1  
@Isaac What is missing from the explanation given at en.wikipedia.org/wiki/… ? –  whuber Nov 26 '10 at 16:51
1  
@Isaac The point of that article is "there is no general rule for the threshold." That is, the answer to your question is you cannot find an optimum $v_0$ analytically because there is nothing to be optimized in the general problem. This situation invites you to examine your data more closely via "analysis of the one-dimensional distribution." In other words, you need more information in order to find a demonstrably best value for$v_0$. –  whuber Nov 26 '10 at 17:27
2  
@Isaac If I have understood the context correctly it seems we "know" nothing: all parameters are estimates from data. Thus, at a minimum, we would want some information about the standard errors of these estimates. If I'm wrong and you know the means and covariance matrix for certain (or have highly accurate estimates), then the problem is one-dimensional: it comes down to separating two normal distributions on the line, which is straightforward to do (this is the basis for power calculations when testing equivalence of means, for instance). Would that be what you're looking for? –  whuber Nov 26 '10 at 19:05
show 3 more comments

1 Answer 1

up vote 3 down vote accepted

When $X$ is normally distributed with known mean $M_1$ and covariance $\Sigma_1$ or with mean $M_2$ and covariance $\Sigma_2$, as indicated in comments to the question, then $V^{\ '}X$ is normally distributed either with mean $\mu_1 = V^{\ '} M_1$ and covariance $\sigma_1^2 = V^{\ '} \Sigma_1 V$ or with mean $\mu_2 = V^{\ '} M_2$ and covariance $\sigma_2^2 = V^{\ '} \Sigma_2 V$; $\mu_2 \gt \mu_1$. We might then care to optimize the chance of correct classification. This can be done provided we stipulate a prior distribution for the two classes. Letting $\pi_1$ be the chance of class 1 and $\pi_2$ the chance of class 2 and $\phi$ the standard normal pdf, then the posterior probabilities of the classes are equal (and therefore $x$ is at the threshold) when

$$f(x) = \pi_1 \phi(\frac{x - \mu_1}{\sigma_1}) - \pi_2 \phi(\frac{x - \mu_2}{\sigma_2}) = 0.$$

There will be at most one zero of $f$ between $x = \mu_1$ and $x = \mu_2$. (When the zeros lie outside this interval we might question the utility of this classifier.) Assuming one exists and choosing $v_0$ to be the negative of this zero gives a linear classifier $X \to V^{\ '}X + v_0$ that, when negative, indicates class 1 is more likely than class 2 and, when positive, indicates class 2 is more likely than class 1.

A simple case arises when the two classes are taken to be equally likely, $\pi_1 = \pi_2 = 1/2,$ for then it is clear from the symmetry and unimodality of $\phi$ that $v_0 = -(\mu_1 + \mu_2)/2$. Note, though, that in general it is not the case that the zero equals $\pi_1 \mu_1 + \pi_2 \mu_2$ (although that might be a good starting guess in a systematic search for the zero).

share|improve this answer
    
After trying to find $v_0$ for this problem with a assumption of $\pi_1 = \pi_2 = 1/2$, i realized that $v_0 = -(\mu_1 + \mu_2)/2$ cannot give me the $v_0$ because the $\mu_i$'s are unknown (although $M_1$ and $M_2$ are known) and they directly depend on $v_0$ value. How i can determine $\mu_1$ and $\mu_2$ ? –  Isaac Nov 28 '10 at 9:09
    
@Isaac When asked, you specifically said you "know [the] underlying distribution parameters for certain"! When you don't know them, you'll have to guess. One approach is to estimate v0, use that to estimate mu1 and mu2 from M1 and M2, then compute the "optimal" v0. Iterate until the value of v0 does not change. –  whuber Nov 28 '10 at 19:29
    
yes, i know the underlying distribution parameters that are $M_1$, $M_2$,$\Sigma_1$, and $\Sigma_2$. But $\mu_1$ $\mu_2$ are not parameters of data distribution, they are parameters of the distribution of $h(x)$ that is a mapping from data to 1-D space. And they depend on $v_0$ value. anyway, I believe a good solution is what you said: iteration until finding a good $v_0$. Then is there really no way to find best threshold value for Fisher criterion analytically? –  Isaac Nov 28 '10 at 20:20
    
@Isaac Sorry, I completely misremembered what is going on here. I should reread my own answers! I defined mu1 and mu2 in the first sentence: if you know M1, M2, and the covariances, that determines V which determines mu1 and mu2. –  whuber Nov 28 '10 at 22:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.