How to make sense of this PCA plot with logistic regression decision boundary (breast cancer data)?

Question

I have a breast cancer data with 13 features. I instantiate PCA and put the training data through and transform down into 2 dimensions.

I get a data that is linearly separable which is interesting to me since I'm doing a binary logistic regression.

I'm writing an article and showing the data with a decision boundary is a good image to show that the model worked.

I train the logistic regression model on the 2-d data from the PCA. I plot the decision boundary using the intercept and coefficient and it does linearly separate the data.

My question is, What does plot of the 2-d data from the PCA and decision boundary mean? What can deduce from that?

Jupyter notebook to my work (It's the very last plot)

---

Edit: Plot with the target class

Plot with predict on x_pca

Answer 1

What does the 2-d PCA data/plot mean?

The 2-d PCA data/plot represent two "compound features" which PCA created to capture as much of the variance in your original 13 features as possible.

Assuming your 13 features are linearly independent (e.g. one feature is not just another feature times 2, for every row in your data), it would take 13 dimensions to capture 100% of the variation in your 13 raw features. However, often PCA can capture e.g. 98% of the variation in your data in just a few PCA dimensions. To see how much of the variance is explained by each PCA dimension for your problem, print x_pca.explained_variance_ratio_ after you fit() your x_pca object.

When PCA can capture a large amount of the variance of your features in just 2 dimensions, that's especially convenient because then you can plot those 2 PCA dimensions as you have, and know that any groupings which show up on the 2-d plot correspond to natural groupings in your 13-dimensional data.

What does the decision boundary mean?

The decision boundary in your code is a prediction of your target variable, using as features (independent variables) the first two PCA dimensions of your 13-dimension original feature set.

Why is your decision boundary not in the obvious gap?

Remember the PCA dimensions were formed just based on your 13 independent variables, without looking at your target. The decision boundary is not a decision boundary between PCA clusters, it's a decision boundary using the PCA dimensions to predict target.

So the fact that the decision boundary is not totally between the clusters means PCA's first two dimensions of your 13 features do a good job of separating your target classes, but not a perfect job.

How to improve your plot

What you really care about is target, right? So in your plot, don't plot all the points as red. Color them by target class. Then you will have a plot that shows you how well PCA and the information in your original features (represented by distance/space between clusters on your plot) distinguishes between target classes (which would be colors on the new plot).

Answer 2

From your question:

I train the logistic regression model on the 2-d data from the PCA. I plot the decision boundary using the intercept and coefficient and it does linearly separate the data.

Logistic regresssion is not a classifier. Its coefficients certainly do not represent a "decision boundary." You have a model that's $$ \log \frac{p}{1-p} = \beta_0 + \beta_1x_1 + \beta_2x_2 $$ Where $p$ is the probability of your outcome, and the $x$-s are your principal components. In the below code, from your notebook, you're using $\beta_0$ and $\beta_1$ as the coefficients to a line in the original predictor space, and it's just dumb luck that it's anywhere near the gap in your data points.

new_model.fit(x_pca, target)
y_intercept = new_model.intercept_    # <- this is beta_0
slope = new_model.coef_[0][0]         # <- this is beta_1
x_axis = np.linspace(-65, 113, 178)

A decision would be based on where $\log p = \log (1-p)$ or some other threshold like that. This is what's not making sense with your plot, and how much variance the PCs capture is a secondary concern.

Assuming you want your decision to be at $\log p = \log (1-p)$, this translates to $$\beta_0 + \beta_1x_1 + \beta_2x_2 = 0.$$

Let's say $x_1$ is the first principal component, and $x_2$ is the second. Then $x_2$ is the $y$-axis in your original plot. Rewriting the above, the boundary should be

$$x_2 = -\frac{\beta_0}{\beta_2} - \frac{\beta_1}{\beta_2}x_1.$$

That is to say that the intercept you want is $-\frac{\beta_0}{\beta_2}$ and the slope is $-\frac{\beta_1}{\beta_2}$.

Note that a such a decision is subjective, and while using $\log p = \log (1-p)$ might fit with your particular view of risk, others might like the raw probability estimate to make their own decision.

Answer 3

@MaxPower has a good answer, and I want to elaborate on his point A) in his comment:

This means either one of two things: either A) your 2 PCA dimensions don't capture enough of the information contained in your raw 13 features, or B) your problem is very hard to predict, even with all the information from your 13 features.

In your question, you don't show how much of the initial 13 variables is represented by the first two principal components. One thing that is easy to forget when doing PCA is the fact that there are more components left than just the first two.

If for instance the 13 original variables are relatively uncorrelated, the first two principle components will only capture a part of the data. The rest will be stored in components 3 through 13.

Why is this relevant?

This is relevant, because your target variable might actually be explained by the third principal component. In that case, you wouldn't be able to see that using the plots you have used now.

What is the takeaway?

Before interpreting the PCA plot of PC1 and PC2, first take a look at the variance explained by these two components. If they together explain a lot (>90%) of the variance in the data, you can quite safely ignore the rest, but if it only explains part of the variance, you should look at the other components as well.

Further remarks

The link to your jupyter notebook is dead, so I can't see exactly what model you used to predict. If you used the entire PCA data, so all 13 principal components, for your prediction, it is likely that your problem falls under B). That means that there more likely wouldn't be a PC3-PC13 that does predict your target well. Because if there was a good predictor, the predicted values in your last plot likely would've been less wrong than they are now.

So either:

• You predicted the target on just PC1 and PC2, which you cannot really do without first checking the cumulative variance explained.

• Your data just does not predict the target well enough.

Another remark:

I get a data that is linearly separable which is interesting to me since I'm doing a binary logistic regression.

I'm writing an article and showing the data with a decision boundary is a good image to show that the model worked.

This data is not linearly separable. At least not in the plots that you show. Yes there are two clear groups, but they are not related to your target variable. Linearly separable would be if you can divide the yellow dots from the purple dots with a linear line. This is not the case here, as you can see that the purple and yellow groups overlap. Furthermore, the decision boundary as is, does little to nothing to actually predict the correct targets, as you can see in your last plot.