Why not always use Polynomial Regression to solve classification problems? [closed]

Question

Consider this simple classification problem:

You can solve it using Logistic Regression. But there's another way. As @whuber noted in this answer, in hypothesis $h(x) = \frac{1}{1 + e^{-P(x)}}$, polynomial $P(x)$ only needs to spit positive numbers for our "$+$" class, and negative numbers for our "$-$" class. In this case the perfect decision boundary will be found.

So why not just apply polynomial regression to this problem without using gradient descent at all? Here's what I mean. Suppose, your features are $x$ and $y$ and you make some polynomial features as follows: $X = \begin{bmatrix} x & y & x^2 & xy & y^2 \end{bmatrix}$ - this is your design matrix. Your desired outcomes are $y = \begin{bmatrix} 1 & 0 & 0 & 1 & ... & 1 \end{bmatrix}^{\mathrm T}$. Instead of using Logistic regression, you can just turn all zeros in $y$ into $-1$ like this: $y = \begin{bmatrix} 1 & -1 & -1 & 1 & ... & 1 \end{bmatrix}^{\mathrm T}$ and apply least squares to $X\theta = y$. By doing so you'll find the polynomial producing positive numbers for "$+$" class and negative numbers for "$-$" class. This equation will have an exact solution whenever $X$ is square. In this case we perfectly separate two classes.

Here's the result of classification after using Logistic Regression:

And here's the result after using Polynomial Regression as I described:

They're just the same. So why not prefer OLS Linear Regression with polynomial features over gradient descent in Logistic Regression? Any disadvantages of my approach? I realize that Polynomial Regression used in such way can lead to overfitting, but let's omit this for a moment.

Answer 1

You can use polynomial fitting to perform classification. However, for your proposed approach there are some issues regarding polynomial regression that logistic avoids.

Firstly, what you are proposing regarding recoding of the dependent variable values from e.g. $\mathbf{y}=(0,1,\ldots,0)$ to $\mathbf{y}=(-1,1,\ldots,-1)$ is not new. This has been applied to classification problems using both multiple linear and multivariate normal regression. Assuming a simple threshold of zero, predicted values are assigned to class 1 if $\hat{y_i}\geq 0$ and 0 if $\hat{y_i}< 0$ -- and the threshold can be optimized for the problem. However, this recoding scheme for $\mathbf{y}$ or $\mathbf{Y}$ when used with straightforward multiple linear regression and multivariate linear regression will only work when the data (classes) are linearly separable. You have shown that non-linear transformations plugged into the linear models does not require linearly separable data.

FYI - I always use linear regression with this recoding scheme for $y$ first, so I know if the data are linear separable. If not, then I gradually will run other classifiers such as Naive Bayes, k-nearest neighbor, linear discriminant, etc., leading up to the computationally-expensive support vector machines and neural networks -- but really only to overcome the problem of the data not being linearly separable in the first place.

Another issue is that, obviously, your approach is a one-step procedure, whereas logistic will use Newton-Raphson iteratively to maximize the log-likelihood. The performance of your approach will also depend specifically on the feature transformations used, whereas for logistic it's not an issue. In other words, if your original features are $x_1, x_2,\ldots,x_p$, you will have to transform and input, for example, $x_1^2, x_1x_2, x_2^2, x_1x_3,\ldots$ into your polynomial regression, whereas for logistic you will input only $x_1, x_2,\ldots,x_p$. Certainly, the two approaches will vary depending on how ill-conditioned the data are, essentially the ratio $\gamma=p/n$, where $p$ is the number of predictors, and $n$ is the number of observations.

An advantage with logistic is that feature selection issues will only depend on the features used, whereas for your approach, feature selection depends on both the features used and the transformations made -- so you'll have twice the work. You can also acquire odds-ratios from logistic, but not your method. In summary, you'll probably be chasing the usefulness of logistic, rather than logistic chasing your new approach.