# Finding the variables that explain a label through LDA and PCA

I have a dataset of about 200 continuous variables with an also continuous target variable. I want to find those predictors that explain target values that are below 50. To do that I create an additional variable in the dataset called "incident" whenever the target value is less than 50.

dataframe['incident'] = np.where(dataframe['valueToPredict']< 50, 'yes', 'no')


And then I perform PCA and LDA.

With PCA I get the first two components which explain about 50% of the variance. Not much, but I want to visualise this components in a 2d plot. I actually plot the different rows of the dataset with the position given by these two first components and colour them by the variable "incident"

As you can see there is a small cluster of incidents in the right. I also perform LDA to separate the two classes with only one component (there are only two clases: "yes" and "no") and plot an histogram of the LDA component.

I want to know if I'm right with the following conclusions and I also have some questions:

• Since the incidents are well separated both in the LDA and PCA plot. Does this mean that I should be able to classify them correctly with the dataset I have?.

• In fact, my problem is not a classification one, but a regression one. The problem I have is that getting the right regressed values whenever there are this kind of abnormal values seems to be impossible with my prediction model. Does this mean that with a classification problem these situations are separable but not with a regression problem?

• How can I get the variables in this dataset of 200 features that work best to separate this abnormal or incident values?. I suppose that the best option is to look at the LDA component and not into PCA, as LDA is optimised to separate two classes.

UPDATE: I attach one more image that could explain further details. There is a predictor that is highly correlated with the value to predict in those cases that are above 50 (blue color). However for those that are under 50 (orange), the same variable does not explain much of the valueToPredict. It is true that there are less samples below 50, but, Could this mean that we are missing a variable in the dataset that explains these kind of values (those < 50)?

Since the incidents are well separated both in the LDA and PCA plot. Does this mean that I should be able to classify them correctly with the dataset I have?.

• The only thing that this "means" is that you have identified two different distributions across both meta-feature spaces (PCA and LDA). Additional meaning comes from defining those distributions (or some boundary between them) and then validating some metric (accuracy, specificity, etc.) across a rigorous set of samples. If your eyeballs and brain can easily separate them, that's great, but you still need to translate that understanding into a model, and that is where your research would begin. You won't know until you know.

In fact, my problem is not a classification one, but a regression one. The problem I have is that getting the right regressed values whenever there are this kind of abnormal values seems to be impossible with my prediction model. Does this mean that with a classification problem these situations are separable but not with a regression problem?

• If you are aiming to regress to a continuous label space, you should reframe your problem. Binary classification performance does not translate to regression. You will know to what degree you can solve this regression problem when you train a model and cross-validate it against a regression label set and metric like mae, rmse, etc.

• You mention "abnormal values". If you mean to say that there are certain observations within your training set that are already known to be derived from a separate process that the one you are attempting to model, then by all means, cull them. If you are saying that you've examined your regression residuals and identified heteroscedasticity (two or more individual distributions), then you need to find the root cause of this heteroscedasticity (this can be a result of improper sampling, testing, or of your training set representing the output of two or more separate, underlying processes) and address it either in your modeling approach or in the features you include in your experiment.

How can I get the variables in this dataset of 200 features that work best to separate this abnormal or incident values?. I suppose that the best option is to look at the LDA component and not into PCA, as LDA is optimised to separate two classes.

• First I must say that you need to understand exactly what makes these observations "abnormal" or "incident". If what makes them so is their high residual within your regression model, then you could just use that feature to classify. You could then visualize these poor performers across your current feature space (and perhaps some expanded feature space if you find no "explanation" in your current features) in search of some set of observations that are outliers from a performance perspective. This will at least tell you on what type of observations your model is struggling to perform.

• High level diagnosis: you seem to have encountered heteroscedastic performance within your regression problem. That much is quite normal and I've seen it myself countless times. I am not sure why you turned to dimensionality reduction techniques such as as LDA or PCA. These will in fact make the root cause of your heteroscedasticity more difficult to identify, and would only make sense if you were having trouble labeling these "outliers" in the first place. However, these "outliers" are by definition those observations that perform poorly within your regression model, and so you already have a label with which to start your root cause analysis.

• High level prognosis: Stop trying to explain your poor regression performance with LDA and PCA. Visualize your regression residuals via histogram and look for heteroscedasticity. If you observe it, then start with some bivariate analysis of your regression residuals across each of your individual features. You can continue with different types of interpretable, regularized regression, but unless you have a prohibitively large set of features (200 isn't), this is overkill. If there is a single feature or some set of features that captures all of the heteroscedasticity (demonstrates multiple distinct distributions within the residual space), then you at least understand what pieces of the feature space the model is struggling to learn. You would then need to either adjust your modeling approach to account for the nature of this feature distribution, train multiple models for each of these distinct processes, or acknowledge that you simply don't have a performant model for one or more of these processes.

• An example: If you are training an image classification model to solve a binary classification problem: animal or not animal? and you feed it all images of either penguins or lions (and of course the necessary negatives), and you examine your accuracy and find heteroscedasticity, a potential hypothesis could be that your model can identify lions well, but not penguins. To validate or deny this, you should look across your "features" of "is lion" or "is penguin" (these would obviously be new features created for this analysis as opposed to fed to the model), and potentially notice distinct distributions within the residual space, letting you know that your model (most likely because of a training or testing error, or perhaps because penguins aren't really that animally when you think about it) just cannot identify penguin, but is fine handling lions.

Man that was so much more than I initially planned on typing, but trying to get you back on track. Best of luck!

Update in response to OP’s update:

The graph you added is indeed a bivariate analysis but, if I understand you correctly, of the target against a certain feature. This “correlation” study does not have meaning as far as identifying heteroscedasticity in the performance of your model.

Your bivariate analyses should analyze either the target or a single feature against the residuals, or error terms, your model produces. This will show you which features (including the meta feature ‘target above 50’”, are connected to heteroscedastic results and is how you can begin to identify the model’s “blind spot(s)”.

You are almost there!

Example: If you adjust your analysis to be of the residuals, as described, and find that “target > 50” enables a linear separation between two distinct error distributions, then you know you need to either engineer a new feature that your current model can work with, change models to something that can understand that meta feature without additional feature engineering, or train a separate model for this separate process that occurs when t > 50.

• Binary classification performance does not translate to regression. What do you mean saying it? LDA with 2 classes, for example, is closely related to linear regression. – ttnphns Jul 23 at 19:59
• @ttnphns I wasn't opining on any "relationship" between those two models. To clarify my point, however, I mean that some classification metric (accuracy, specificity, etc.) has no direct relationship with any regression metric (mae, rmse, etc.) and so he/she cannot imply that the LDA's "effectiveness" will translate to some level of performance in a regression setting – John H Jul 23 at 20:11
• You said I should focus on abnormal or incident observations and that I should visualise these poor performers across my feature space. But how can I visualise this with a feature space of 200 features?. That's why I turned to PCA and LDA; to be able to visualise that feature space in a simpler form. I suppose the answer is, as you said: "Visualize your regression residuals via histogram and look for heteroscedasticity". If I understand correctly I should separate those samples that have high residuals and do a bivariate plot like these en.wikipedia.org/wiki/Bivariate_analysis. – Brandon Jul 24 at 5:51
• @Brandon, 200 plots really should not be that difficult to produce. However, another option is to train some sort of interpretable, regularized model like a lasso or ridge regression with your residual as the target, and then examine your coefficients as a place to start your bivariate analysis – John H Jul 24 at 13:32
• @Brandon there also isn’t a need to “separate” any points from any others before examining your residuals via histogram. You want everything present so that you can understand the nature of the heteroscedasticity. Unless it’s clear that the heteroscedasticity is the result of some additional process that you unintentionally sampled from, you should try to remedy it by adding/engineering additional features or choosing a model that can model both/all processes efficiently at the same time. Otherwise you are creating a testing bias. – John H Jul 24 at 13:38