Formula for decision boundary of a classifier (in order to visualize it)

Question

I'm confused on how to plot decision boundary for classifiers.

For example, i'm working with perceptron. So, the formula for decision boundary(if I understand this correctly) is

W1x + W2y + W_bias = 0

It's equal 0 because (again, if i understand this right): the activation function is +1 if the dot product of W and x >0 and -1 if otherwise. This makes the decision boundary equals 0. Is this right?

While this is simple for perceptron, what is the formula for decision boundary logistic regression? It can't be

sigmoid(W1x) + sigmoid(W2x) + W3 = 0

can it?

How do I determine decision boundary formula for logistic regression or any other classifier (particularly nonlinear ones)?

Answer 1

For logistic regression, it's actually \begin{gather} p(x) = \frac{1}{1 + \exp(- (\sum_j W_j x_j + b) )} = \frac12 \\ 1 + \exp(- (\sum_j W_j x_j + b) ) = 2 \\ \exp(- (\sum_j W_j x_j + b) ) = 1 \\ - (\sum_j W_j x_j + b) = 0 \\ \sum_j W_j x_j + b = 0 ,\end{gather} exactly the same as for the perceptron.

In general, you can try to solve for $p(x) = \frac12$ when you get a probability estimate (and the prior is bounded), or more generally if your prediction is $\mathrm{sign}(f(x))$, for $f(x) = 0$. But for nonlinear classifiers, this may be difficult to solve, and thus it's often easier to just plot the output of the classifier on a fine grid as Nick suggested.

Answer 2

This is, to me, a matter of taste. In the past what I've done (and what I'd recommend) is to grid your space into a fine grid and run the points on the grid through your classifier. You can then color those points with the given class and plotting those colored points will give you a visual decision bound if your grid is fine enough. Tibshirani and Hastie have some code in their online ESL book for doing that