@MaxPower has a good answer, and I want to elaborate on his point A) in his comment:
This means either one of two things: either A) your 2 PCA dimensions
don't capture enough of the information contained in your raw 13
features, or B) your problem is very hard to predict, even with all
the information from your 13 features.
In your question, you don't show how much of the initial 13 variables is represented by the first two principal components. One thing that is easy to forget when doing PCA is the fact that there are more components left than just the first two.
If for instance the 13 original variables are relatively uncorrelated, the first two principle components will only capture a part of the data. The rest will be stored in components 3 through 13.
Why is this relevant?
This is relevant, because your target variable might actually be explained by the third principal component. In that case, you wouldn't be able to see that using the plots you have used now.
What is the takeaway?
Before interpreting the PCA plot of PC1 and PC2, first take a look at the variance explained by these two components. If they together explain a lot (>90%
) of the variance in the data, you can quite safely ignore the rest, but if it only explains part of the variance, you should look at the other components as well.
Further remarks
The link to your jupyter notebook is dead, so I can't see exactly what model you used to predict. If you used the entire PCA data, so all 13 principal components, for your prediction, it is likely that your problem falls under B). That means that there more likely wouldn't be a PC3-PC13 that does predict your target well. Because if there was a good predictor, the predicted values in your last plot likely would've been less wrong than they are now.
So either:
You predicted the target on just PC1 and PC2, which you cannot really do without first checking the cumulative variance explained.
Your data just does not predict the target well enough.
Another remark:
I get a data that is linearly separable which is interesting to me since I'm doing a binary logistic regression.
I'm writing an article and showing the data with a decision boundary
is a good image to show that the model worked.
This data is not linearly separable. At least not in the plots that you show. Yes there are two clear groups, but they are not related to your target variable. Linearly separable would be if you can divide the yellow dots from the purple dots with a linear line. This is not the case here, as you can see that the purple and yellow groups overlap. Furthermore, the decision boundary as is, does little to nothing to actually predict the correct targets, as you can see in your last plot.
target
classes (based on PCA dimensions as features), not to separate PCA clusters. The PCA dimensions/clusters themselves are created without knowing thetarget
directly, although we hope they correspond to differences intarget
. See my answer below for more explanation. $\endgroup$