5
$\begingroup$

I have a breast cancer data with 13 features. I instantiate PCA and put the training data through and transform down into 2 dimensions.

I get a data that is linearly separable which is interesting to me since I'm doing a binary logistic regression.

I'm writing an article and showing the data with a decision boundary is a good image to show that the model worked.

I train the logistic regression model on the 2-d data from the PCA. I plot the decision boundary using the intercept and coefficient and it does linearly separate the data.

enter image description here

My question is, What does plot of the 2-d data from the PCA and decision boundary mean? What can deduce from that?

Jupyter notebook to my work (It's the very last plot)


Edit: Plot with the target class

enter image description here

Plot with predict on x_pca

enter image description here

$\endgroup$
5
  • 2
    $\begingroup$ Why is your decision boundary not in the obvious gap? $\endgroup$
    – Henry
    Commented May 1, 2017 at 11:09
  • $\begingroup$ I noticed and I've got no idea. $\endgroup$
    – MAA
    Commented May 1, 2017 at 11:15
  • $\begingroup$ it's because the decision boundary is a decision boundary using PCA dimensions to separate/predict target classes (based on PCA dimensions as features), not to separate PCA clusters. The PCA dimensions/clusters themselves are created without knowing the target directly, although we hope they correspond to differences in target. See my answer below for more explanation. $\endgroup$
    – Max Power
    Commented May 1, 2017 at 11:37
  • 1
    $\begingroup$ The link to your notebook is dead. $\endgroup$
    – JAD
    Commented May 2, 2017 at 6:29
  • $\begingroup$ @JarkoDubbeldam Fixed it. $\endgroup$
    – MAA
    Commented May 2, 2017 at 6:51

3 Answers 3

4
$\begingroup$

What does the 2-d PCA data/plot mean?

The 2-d PCA data/plot represent two "compound features" which PCA created to capture as much of the variance in your original 13 features as possible.

Assuming your 13 features are linearly independent (e.g. one feature is not just another feature times 2, for every row in your data), it would take 13 dimensions to capture 100% of the variation in your 13 raw features. However, often PCA can capture e.g. 98% of the variation in your data in just a few PCA dimensions. To see how much of the variance is explained by each PCA dimension for your problem, print x_pca.explained_variance_ratio_ after you fit() your x_pca object.

When PCA can capture a large amount of the variance of your features in just 2 dimensions, that's especially convenient because then you can plot those 2 PCA dimensions as you have, and know that any groupings which show up on the 2-d plot correspond to natural groupings in your 13-dimensional data.

What does the decision boundary mean?

The decision boundary in your code is a prediction of your target variable, using as features (independent variables) the first two PCA dimensions of your 13-dimension original feature set.

Why is your decision boundary not in the obvious gap?

Remember the PCA dimensions were formed just based on your 13 independent variables, without looking at your target. The decision boundary is not a decision boundary between PCA clusters, it's a decision boundary using the PCA dimensions to predict target.

So the fact that the decision boundary is not totally between the clusters means PCA's first two dimensions of your 13 features do a good job of separating your target classes, but not a perfect job.

How to improve your plot

What you really care about is target, right? So in your plot, don't plot all the points as red. Color them by target class. Then you will have a plot that shows you how well PCA and the information in your original features (represented by distance/space between clusters on your plot) distinguishes between target classes (which would be colors on the new plot).

$\endgroup$
6
  • $\begingroup$ Truly a wonderful answer. I've tried to color the points on the plot using the target class. Unfortunately, it doesn't seem to me that it's doing well when I do that. What about using "predict" on the x_pca data? Would that mean anything? $\endgroup$
    – MAA
    Commented May 1, 2017 at 12:42
  • 1
    $\begingroup$ I will edit the question and I will include a picture showing the colors using the target class and the prediction of "new_model" on the "x_pca" data. $\endgroup$
    – MAA
    Commented May 1, 2017 at 12:42
  • $\begingroup$ And my legend is messed up now. Plotting is hard :) $\endgroup$
    – MAA
    Commented May 1, 2017 at 13:15
  • 2
    $\begingroup$ so the first new plot you added suggests to me that your two clusters don't really separate your target classes, since there's a lot of both colors on either side of your decision boundary. This means either one of two things: either A) your 2 PCA dimensions don't capture enough of the information contained in your raw 13 features, or B) your problem is very hard to predict, even with all the information from your 13 features. $\endgroup$
    – Max Power
    Commented May 1, 2017 at 13:31
  • $\begingroup$ I suspect its A. I'll not touch on PCA in my article then. Thank you Max Power. You've been awesome. $\endgroup$
    – MAA
    Commented May 1, 2017 at 14:08
4
$\begingroup$

From your question:

I train the logistic regression model on the 2-d data from the PCA. I plot the decision boundary using the intercept and coefficient and it does linearly separate the data.

Logistic regresssion is not a classifier. Its coefficients certainly do not represent a "decision boundary." You have a model that's $$ \log \frac{p}{1-p} = \beta_0 + \beta_1x_1 + \beta_2x_2 $$ Where $p$ is the probability of your outcome, and the $x$-s are your principal components. In the below code, from your notebook, you're using $\beta_0$ and $\beta_1$ as the coefficients to a line in the original predictor space, and it's just dumb luck that it's anywhere near the gap in your data points.

new_model.fit(x_pca, target)
y_intercept = new_model.intercept_    # <- this is beta_0
slope = new_model.coef_[0][0]         # <- this is beta_1
x_axis = np.linspace(-65, 113, 178)

A decision would be based on where $\log p = \log (1-p)$ or some other threshold like that. This is what's not making sense with your plot, and how much variance the PCs capture is a secondary concern.

Assuming you want your decision to be at $\log p = \log (1-p)$, this translates to $$\beta_0 + \beta_1x_1 + \beta_2x_2 = 0.$$

Let's say $x_1$ is the first principal component, and $x_2$ is the second. Then $x_2$ is the $y$-axis in your original plot. Rewriting the above, the boundary should be

$$x_2 = -\frac{\beta_0}{\beta_2} - \frac{\beta_1}{\beta_2}x_1.$$

That is to say that the intercept you want is $-\frac{\beta_0}{\beta_2}$ and the slope is $-\frac{\beta_1}{\beta_2}$.

Note that a such a decision is subjective, and while using $\log p = \log (1-p)$ might fit with your particular view of risk, others might like the raw probability estimate to make their own decision.

$\endgroup$
7
  • $\begingroup$ Thanks for highlighting "Logistic regresssion is not a classifier. Its coefficients certainly do not represent a "decision boundary." ". This seems to be a significant omission from my answer $\endgroup$
    – Max Power
    Commented May 2, 2017 at 12:05
  • $\begingroup$ @MaxPower Not sure if I am missing some details, but logistic regression can be used for classification, so I don't really see where that statement comes from. $\endgroup$
    – JAD
    Commented May 2, 2017 at 12:08
  • $\begingroup$ Based on your text at the end, are you saying the log reg line would imply a decision boundary if your decision threshold was +/- 50% based on the probability output by the logistic regression? But that the line would shift for other plausible probability thresholds? $\endgroup$
    – Max Power
    Commented May 2, 2017 at 12:08
  • $\begingroup$ @MaxPower what other probability thresholds would be plausible? $\endgroup$
    – JAD
    Commented May 2, 2017 at 12:10
  • $\begingroup$ @MaxPower The logistic regression line itself describes odds in the log space. The decision threshold would be a point somewhere on this line, above which everything would be considered "true". When you set the decision threshold at $p=.5$ you're saying that the cost of misclassifying a true is the same as that of misclassifying a false. This is a value-laden statement that may or may not be correct; I'd argue that in many cases it's not. $\endgroup$
    – einar
    Commented May 2, 2017 at 12:26
2
$\begingroup$

@MaxPower has a good answer, and I want to elaborate on his point A) in his comment:

This means either one of two things: either A) your 2 PCA dimensions don't capture enough of the information contained in your raw 13 features, or B) your problem is very hard to predict, even with all the information from your 13 features.

In your question, you don't show how much of the initial 13 variables is represented by the first two principal components. One thing that is easy to forget when doing PCA is the fact that there are more components left than just the first two.

If for instance the 13 original variables are relatively uncorrelated, the first two principle components will only capture a part of the data. The rest will be stored in components 3 through 13.

Why is this relevant?

This is relevant, because your target variable might actually be explained by the third principal component. In that case, you wouldn't be able to see that using the plots you have used now.

What is the takeaway?

Before interpreting the PCA plot of PC1 and PC2, first take a look at the variance explained by these two components. If they together explain a lot (>90%) of the variance in the data, you can quite safely ignore the rest, but if it only explains part of the variance, you should look at the other components as well.

Further remarks

The link to your jupyter notebook is dead, so I can't see exactly what model you used to predict. If you used the entire PCA data, so all 13 principal components, for your prediction, it is likely that your problem falls under B). That means that there more likely wouldn't be a PC3-PC13 that does predict your target well. Because if there was a good predictor, the predicted values in your last plot likely would've been less wrong than they are now.

So either:

  • You predicted the target on just PC1 and PC2, which you cannot really do without first checking the cumulative variance explained.

  • Your data just does not predict the target well enough.

Another remark:

I get a data that is linearly separable which is interesting to me since I'm doing a binary logistic regression.

I'm writing an article and showing the data with a decision boundary is a good image to show that the model worked.

This data is not linearly separable. At least not in the plots that you show. Yes there are two clear groups, but they are not related to your target variable. Linearly separable would be if you can divide the yellow dots from the purple dots with a linear line. This is not the case here, as you can see that the purple and yellow groups overlap. Furthermore, the decision boundary as is, does little to nothing to actually predict the correct targets, as you can see in your last plot.

$\endgroup$
5
  • $\begingroup$ I've fixed the jupyter notebook link. I think the first two components relatively explain the data well. Although the third component is not that far behind. However, I don't know how to interpret the "explained_variance_" variable. $\endgroup$
    – MAA
    Commented May 2, 2017 at 7:34
  • $\begingroup$ Here is the explained variance for the first 4 components. How do I interpret this? [ 940.43154588, 231.08268744, 119.42056463, 18.97322897] $\endgroup$
    – MAA
    Commented May 2, 2017 at 7:35
  • $\begingroup$ The total variance of the data is the sum of the variances of all the components. If you run PCA(n_components=None) you will get all the components as output. The sum of their variances is the total variance. What you want to look at, is $\sum_{i=1}^k(Var(PC_i)/totalVar)$. By the looks of the first 4 values, this would be $76\%$ for PC1, $89\%$ for PC1 and 2, and $98\%$ for PC 1-3. This means that if you reduce the data to the first principal components, you keep $89\%$ of your data. This is pretty reasonable. ... $\endgroup$
    – JAD
    Commented May 2, 2017 at 8:17
  • $\begingroup$ You could say that the majority of the information in your data is contained in the first two (or three) principal components. $\endgroup$
    – JAD
    Commented May 2, 2017 at 8:18
  • 2
    $\begingroup$ As mentioned in my answer, I suggest using x_pca.explained_variance_ratio_ (not explained _variance) so the numbers come out already normalized as %s of the total variance of your data. From the numbers you posted though, you're clearly losing a lot of information by dropping the third component, although 4-13 would provide little additional info. Since PCA doesn't provide extra info, and your motivation seemed to be getting a 2D plot, this seems to not be a great use case for PCA (since so much of the variance is explained by the third PCA dimension) $\endgroup$
    – Max Power
    Commented May 2, 2017 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.