4
$\begingroup$

I am trying to find a solution to the decision boundary in QDA. The question was already asked and answered for LDA, and the solution provided by amoeba to compute this using the "standard Gaussian way" worked well. However, I am applying the same technique for a 2 class, 2 feature QDA and am having trouble. Would someone be able to check my work and let me know if this approach is correct?

I start-off with the discriminant equation,
$\delta_l = -\frac{1}{2}\log{|\mathbf{\Sigma_i}|}-\frac{1}{2}{\mathbf{(x-\mu_i)'\Sigma^{-1}_i(x - \mu_i)}}+\log{p_i}$

Where $\delta_l$ is the discriminant score for some observation $\mathbf{x}$ belonging to class $l$ which could be 0 or 1 in this 2 class problem.

The decision boundary between $l=0$ and $l=1$ is the vector $\boldsymbol{\vec{x}}$ that satisfies the criteria $\delta_0$ equal to $\delta_1$. I approach this in the following way:

$$\delta_0=\delta_1$$

Substitute the discriminant equation for both $\delta_0$ and $\delta_1$

$$-\frac{1}{2}\log{|\mathbf{\Sigma_0}|}-\frac{1}{2}{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x - \mu_0)}}+\log{p_0} = -\frac{1}{2}\log{|\mathbf{\Sigma_1}|}-\frac{1}{2}{\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x - \mu_1)}}+\log{p_1}$$

Algebraic manipulations...

$$\frac{1}{2}{\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x - \mu_1)}}-\frac{1}{2}{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x - \mu_0)}} = \frac{1}{2}\log{|\mathbf{\Sigma_0}|}-\frac{1}{2}\log{|\mathbf{\Sigma_1}|}+\log{p_1}-\log{p_0}$$

$$\frac{1}{2}({\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x - \mu_1)}}-{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x - \mu_0)}}) = \frac{1}{2}\log{|\mathbf{\Sigma_0}|}-\frac{1}{2}\log{|\mathbf{\Sigma_1}|}+\log{p_1}-\log{p_0}$$

$${\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x - \mu_1)}}-{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x - \mu_0)}} = \log{|\mathbf{\Sigma_0}|}-\log{|\mathbf{\Sigma_1}|}+2\log{p_1}-2\log{p_0}$$

The right side of the above equation is a constant that we can assign to the variable $C$ as follows:

$C = \log{|\mathbf{\Sigma_0}|}-\log{|\mathbf{\Sigma_1}|}+2\log{p_1}-2\log{p_0}$

at this point we have:

$${\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x - \mu_1)}}-{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x - \mu_0)}}=C$$

Next I am trying to solve for the value of y (e.g., feature 2) given some input value of x (feature 1).

To simplify the manipulations, I have temporarily assigned the following variables as: $$x_0 = x-\mu_{00}$$ $$y_0 = y-\mu_{01}$$ $$x_1 = x-\mu_{10}$$ $$y_1 = y-\mu_{11}$$

$$\begin{bmatrix} x_1 & y_1 \\ \end{bmatrix} \begin{bmatrix} a & b \\ c & d \\ \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \\ \end{bmatrix} - \begin{bmatrix} x_0 & y_0 \\ \end{bmatrix} \begin{bmatrix} p & q \\ r & s \\ \end{bmatrix} \begin{bmatrix} x_0 \\ y_0 \\ \end{bmatrix} = C$$
$$x_1(ax_1+by_1) + y_1(cx_1+dy_1)-x_0(px_0+qy_0)-y_0(rx_0+sy_0) = C$$
$$ax^2_1+bx_1y_1+cx_1y_1+dy^2_1-px^2_0-qx_0y_0-rx_0y_0-sy^2_0 = C$$
$$bx_1y_1+cx_1y_1+dy^2_1-qx_0y_0-rx_0y_0-sy^2_0 = C-ax^2_1+px^2_0$$
$$dy^2_1-sy^2_0+bx_1y_1+cx_1y_1-qx_0y_0-rx_0y_0 = C-ax^2_1+px^2_0$$
$$dy^2_1-sy^2_0+x_1y_1(b+c)+x_0y_0(-q-r) = C-ax^2_1+px^2_0$$
substituting for $x_0, y_0, x_1, y_1$ we now have the following: $$d(y-\mu_{11})^2-s( y-\mu_{01})^2+(x-\mu_{10})(y-\mu_{11})(b+c)+(x-\mu_{00})(y-\mu_{01})(-q-r) = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2$$

then I calculated the squares and reduced the terms to the following result: $$(d-s)y^2+(-2d\mu_{11}+2s\mu_{01}+bx-b\mu_{10}+cx-c\mu_{10}-qx+q\mu_{00}-rx+r\mu_{00})y = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2+b\mu_{11}x+c\mu_{11}x-q\mu_{01}x-r\mu_{01}x+d\mu_{11}^2-s\mu_{01}^2-b\mu_{10}\mu_{11}-c\mu_{10}\mu_{11}+q\mu_{01}\mu_{00}+r\mu_{01}\mu_{00}$$
Finally, I can apply the quadratic formula to solve for $y$ where
$u = d-s$
$v = -2d\mu_{11}+2s\mu_{01}+bx-b\mu_{10}+cx-c\mu_{10}-qx+q\mu_{00}-rx+r\mu_{00}$
$w = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2+b\mu_{11}x+c\mu_{11}x-q\mu_{01}x-r\mu_{01}x+d\mu_{11}^2-s\mu_{01}^2-b\mu_{10}\mu_{11}-c\mu_{10}\mu_{11}+q\mu_{01}\mu_{00}+r\mu_{01}\mu_{00}$

The quadratic formula with these variables would be the following: $$y = \frac{-v\pm\sqrt{v^2-4uw}}{2u}$$

After attempting to check this solution on a simple data set I obtain poor results. I cannot figure out if it's the approach to the solution or if something is wrong in my code.

Thanks for your time and help!

$\endgroup$
  • $\begingroup$ there will be plus sign inside the square root in the final roots that you computed which will solve the problem $\endgroup$ – Ankur Sharma Feb 11 '18 at 19:19
  • $\begingroup$ It would be much better if you provided a fuller explanation; this requires a lot of work on the reader to check, and in fact without going to a lot of work I can't see why it would be true. $\endgroup$ – jbowman Feb 11 '18 at 20:36
  • $\begingroup$ This is a weak answer. It does not speak to the question, the method, the motivation. Could you be more clear, or systematic. There are guides about what constitutes a fair answer, and this meets none of those. $\endgroup$ – EngrStudent Feb 11 '18 at 20:56
  • $\begingroup$ Please expand your answer so that it clearly explains your reasoning. $\endgroup$ – Glen_b Feb 11 '18 at 21:44
0
$\begingroup$

If you look at the calculations, you will see there are a few bugs in this.

Correct value of w comes out to be : $$w = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2+b\mu_{11}x+c\mu_{11}x-q\mu_{01}x-r\mu_{01}x-d\mu_{11}^2+s\mu_{01}^2-b\mu_{10}\mu_{11}-c\mu_{10}\mu_{11}+q\mu_{01}\mu_{00}+r\mu_{01}\mu_{00} $$

After then the value of y comes out to be: $$ y = \frac{-v\pm\sqrt{v^2+4uw}}{2u} $$

After making these two changes, you will get the correct quadratic boundary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.