How is the decision boundary's equation determined?

Question

Machine learning newbie/enthusiast here, following Andrew Ng's course @ Coursera.

I'm studying logistic regression. I've understood that, depending on the data, the decision boundary can be described by a simple linear equation (a line) as well as more complex higher-order polynomials (curves, circles, odd shapes).

What's not 100% clear to me: how is the decision boundary's equation determined? For example:

The decision boundary here is a circle, defined as

$ \theta_0 + \theta_1x_1 + \theta_2x_2 + \theta_3x_1^2 + \theta_4x_2^2 = 0 $

Is the logistic regression algorithm capable of generating such a complex formula, or is it something to tweak manually?

Answer 1

You need one additional piece of information to determine a decision boundary: a level to threshold the probabilities. Given a threshold $T$, we make positive decisions when

$$ g(\theta_0 + \theta_1 x_1 + \theta_2 x_2 + \theta_3 x_1^2 + \theta_4 x_2^2) \geq T $$

and negative decisions when

$$ g(\theta_0 + \theta_1 x_1 + \theta_2 x_2 + \theta_3 x_1^2 + \theta_4 x_2^2) < T $$

so the boundary is given by

$$ g(\theta_0 + \theta_1 x_1 + \theta_2 x_2 + \theta_3 x_1^2 + \theta_4 x_2^2) = T $$

In your case, logistic regression, $g$ is the sigmoid function, whose inverse is the log odds, so the decision boundary is

$$ \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \theta_3 x_1^2 + \theta_4 x_2^2 = \log \left(\frac{T}{1-T}\right) $$

The right hand side is just a constant. You can complete the square to figure out what type of geometric curve this determines in any given case.

Andrew got $0$ on the right hand side by setting $T = 0.5$, something I generally would not advise without studying the specific problem you are trying to solve. Thresholds are best set by examining the cost tradeoffs between false negatives and false positives for various values of $T$.

However it's still not clear to me: did Andrew say "Cool, my data can be separated by a circle, let's go with the circle equation [...]"? Did the algorithm figure it out instead?

In this case, certainly the first thing!

Logistic regression has no built in ability to create and use transformations of raw features, and it's common to use exploratory data analysis to assist when building models.

Other approaches are:

• Use a basis expansion of features in the regression, like cubic splines. This will allow the regression to fit very general shapes.
• Use a generalized version of logistic regression, like gradient boosted logistic regression. This has the ability to adaptively create new features to fit your data.

But for a first shot at logistic regression, it's good practice to look at data, and engineer appropriate features. This is almost certainly the lesson Andrew is trying to communicate.