2
$\begingroup$

Consider this simple classification problem:

enter image description here

You can solve it using Logistic Regression. But there's another way. As @whuber noted in this answer, in hypothesis $h(x) = \frac{1}{1 + e^{-P(x)}}$, polynomial $P(x)$ only needs to spit positive numbers for our "$+$" class, and negative numbers for our "$-$" class. In this case the perfect decision boundary will be found.

So why not just apply polynomial regression to this problem without using gradient descent at all? Here's what I mean. Suppose, your features are $x$ and $y$ and you make some polynomial features as follows: $X = \begin{bmatrix} x & y & x^2 & xy & y^2 \end{bmatrix}$ - this is your design matrix. Your desired outcomes are $y = \begin{bmatrix} 1 & 0 & 0 & 1 & ... & 1 \end{bmatrix}^{\mathrm T}$. Instead of using Logistic regression, you can just turn all zeros in $y$ into $-1$ like this: $y = \begin{bmatrix} 1 & -1 & -1 & 1 & ... & 1 \end{bmatrix}^{\mathrm T}$ and apply least squares to $X\theta = y$. By doing so you'll find the polynomial producing positive numbers for "$+$" class and negative numbers for "$-$" class. This equation will have an exact solution whenever $X$ is square. In this case we perfectly separate two classes.

Here's the result of classification after using Logistic Regression:

enter image description here

And here's the result after using Polynomial Regression as I described:

enter image description here

They're just the same. So why not prefer OLS Linear Regression with polynomial features over gradient descent in Logistic Regression? Any disadvantages of my approach? I realize that Polynomial Regression used in such way can lead to overfitting, but let's omit this for a moment.

$\endgroup$
7
  • 3
    $\begingroup$ Why “over gradient descent”? Using gradient descent is completely orthogonal to polynomial regression. You can use gradient descent for training polynomial regression. $\endgroup$
    – Tim
    Nov 11 '21 at 19:33
  • 1
    $\begingroup$ How do you turn the linear regression into a probability predictor? // Logistic regression can use polynomial features, too. $\endgroup$
    – Dave
    Nov 11 '21 at 19:40
  • 4
    $\begingroup$ Please research our posts on "overfitting." That's the essence of the question--there's no dodging that. Also, the confusion between a model (OLS) and a numerical procedure (gradient descent) makes this a particularly murky question. $\endgroup$
    – whuber
    Nov 11 '21 at 19:40
  • 3
    $\begingroup$ Given the length of the discussion alone, your question is rather unclear. You are mixing OLS, polynomial regression, logistic regression, and gradient decent. Those are completely different concepts. It’s hard to understand what exactly you want to compare to what and what’s the actual problem. Please edit to clarify and focus on a specific problem. $\endgroup$
    – Tim
    Nov 11 '21 at 20:29
  • 2
    $\begingroup$ What happens when two observations have the same features but belong to different categories? $\endgroup$
    – Dave
    Nov 11 '21 at 21:05
4
$\begingroup$

You can use polynomial fitting to perform classification. However, for your proposed approach there are some issues regarding polynomial regression that logistic avoids.

Firstly, what you are proposing regarding recoding of the dependent variable values from e.g. $\mathbf{y}=(0,1,\ldots,0)$ to $\mathbf{y}=(-1,1,\ldots,-1)$ is not new. This has been applied to classification problems using both multiple linear and multivariate normal regression. Assuming a simple threshold of zero, predicted values are assigned to class 1 if $\hat{y_i}\geq 0$ and 0 if $\hat{y_i}< 0$ -- and the threshold can be optimized for the problem. However, this recoding scheme for $\mathbf{y}$ or $\mathbf{Y}$ when used with straightforward multiple linear regression and multivariate linear regression will only work when the data (classes) are linearly separable. You have shown that non-linear transformations plugged into the linear models does not require linearly separable data.

FYI - I always use linear regression with this recoding scheme for $y$ first, so I know if the data are linear separable. If not, then I gradually will run other classifiers such as Naive Bayes, k-nearest neighbor, linear discriminant, etc., leading up to the computationally-expensive support vector machines and neural networks -- but really only to overcome the problem of the data not being linearly separable in the first place.

Another issue is that, obviously, your approach is a one-step procedure, whereas logistic will use Newton-Raphson iteratively to maximize the log-likelihood. The performance of your approach will also depend specifically on the feature transformations used, whereas for logistic it's not an issue. In other words, if your original features are $x_1, x_2,\ldots,x_p$, you will have to transform and input, for example, $x_1^2, x_1x_2, x_2^2, x_1x_3,\ldots$ into your polynomial regression, whereas for logistic you will input only $x_1, x_2,\ldots,x_p$. Certainly, the two approaches will vary depending on how ill-conditioned the data are, essentially the ratio $\gamma=p/n$, where $p$ is the number of predictors, and $n$ is the number of observations.

An advantage with logistic is that feature selection issues will only depend on the features used, whereas for your approach, feature selection depends on both the features used and the transformations made -- so you'll have twice the work. You can also acquire odds-ratios from logistic, but not your method. In summary, you'll probably be chasing the usefulness of logistic, rather than logistic chasing your new approach.

$\endgroup$
4
  • 1
    $\begingroup$ Thank you for the answer! But what do you mean by "whereas for your approach, feature selection depends on both the features used and the transformations made"? You can always just raise all your features to 1, 2, 3, ... powers. And this is going to be enough for perfect decision boundary to exist, provided you added enough such features (to make matrix $X$ square). $\endgroup$
    – mathgeek
    Nov 11 '21 at 20:41
  • 2
    $\begingroup$ You can predict anything perfectly if a thousand predictors are used. However, many won't be significant, since their standard errors will large -- but the model will fit. At that point, you'd also be using noise for prediction. Many of your transformed input features may be non-informative, i.e., don't help. For logistic, you'd only need to find out which of the $p$ predictors are non-informative. For your approach, if you use powers of 2,3 and cross-multiples like $xy$ and $(xy)^2$ you'll need to evaluate informativeness of e.g 2-4 times as many predictors. $\endgroup$ Nov 11 '21 at 20:59
  • 2
    $\begingroup$ There's a lot of good and pertinent advice in this answer. But I'm curious about the remark "The performance of your approach will also depend specifically on the feature transformations used, whereas for logistic it's not an issue." Why not? Are you perhaps referring only to the perfect-separation case here? $\endgroup$
    – whuber
    Nov 11 '21 at 21:31
  • $\begingroup$ @whuber - for logistic, performance depends on the input predictors used assuming the transformations involving powers and cross-products are not made. For the proposed approach: there's an additional level of complexity beyond the original feature selection --> the transformations applied to the selected features. $\endgroup$ Nov 11 '21 at 21:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.