Answers to your questions:
Since the incidents are well separated both in the LDA and PCA plot. Does this mean that I should be able to classify them correctly with the dataset I have?.
- The only thing that this "means" is that you have identified two different distributions across both meta-feature spaces (PCA and LDA). Additional meaning comes from defining those distributions (or some boundary between them) and then validating some metric (accuracy, specificity, etc.) across a rigorous set of samples. If your eyeballs and brain can easily separate them, that's great, but you still need to translate that understanding into a model, and that is where your research would begin. You won't know until you know.
In fact, my problem is not a classification one, but a regression one. The problem I have is that getting the right regressed values whenever there are this kind of abnormal values seems to be impossible with my prediction model. Does this mean that with a classification problem these situations are separable but not with a regression problem?
If you are aiming to regress to a continuous label space, you should reframe your problem. Binary classification performance does not translate to regression. You will know to what degree you can solve this regression problem when you train a model and cross-validate it against a regression label set and metric like mae, rmse, etc.
You mention "abnormal values". If you mean to say that there are certain observations within your training set that are already known to be derived from a separate process that the one you are attempting to model, then by all means, cull them. If you are saying that you've examined your regression residuals and identified heteroscedasticity (two or more individual distributions), then you need to find the root cause of this heteroscedasticity (this can be a result of improper sampling, testing, or of your training set representing the output of two or more separate, underlying processes) and address it either in your modeling approach or in the features you include in your experiment.
How can I get the variables in this dataset of 200 features that work best to separate this abnormal or incident values?. I suppose that the best option is to look at the LDA component and not into PCA, as LDA is optimised to separate two classes.
First I must say that you need to understand exactly what makes these observations "abnormal" or "incident". If what makes them so is their high residual within your regression model, then you could just use that feature to classify. You could then visualize these poor performers across your current feature space (and perhaps some expanded feature space if you find no "explanation" in your current features) in search of some set of observations that are outliers from a performance perspective. This will at least tell you on what type of observations your model is struggling to perform.
High level diagnosis: you seem to have encountered heteroscedastic performance within your regression problem. That much is quite normal and I've seen it myself countless times. I am not sure why you turned to dimensionality reduction techniques such as as LDA or PCA. These will in fact make the root cause of your heteroscedasticity more difficult to identify, and would only make sense if you were having trouble labeling these "outliers" in the first place. However, these "outliers" are by definition those observations that perform poorly within your regression model, and so you already have a label with which to start your root cause analysis.
High level prognosis: Stop trying to explain your poor regression performance with LDA and PCA. Visualize your regression residuals via histogram and look for heteroscedasticity. If you observe it, then start with some bivariate analysis of your regression residuals across each of your individual features. You can continue with different types of interpretable, regularized regression, but unless you have a prohibitively large set of features (200 isn't), this is overkill. If there is a single feature or some set of features that captures all of the heteroscedasticity (demonstrates multiple distinct distributions within the residual space), then you at least understand what pieces of the feature space the model is struggling to learn. You would then need to either adjust your modeling approach to account for the nature of this feature distribution, train multiple models for each of these distinct processes, or acknowledge that you simply don't have a performant model for one or more of these processes.
An example: If you are training an image classification model to solve a binary classification problem: animal or not animal? and you feed it all images of either penguins or lions (and of course the necessary negatives), and you examine your accuracy and find heteroscedasticity, a potential hypothesis could be that your model can identify lions well, but not penguins. To validate or deny this, you should look across your "features" of "is lion" or "is penguin" (these would obviously be new features created for this analysis as opposed to fed to the model), and potentially notice distinct distributions within the residual space, letting you know that your model (most likely because of a training or testing error, or perhaps because penguins aren't really that animally when you think about it) just cannot identify penguin, but is fine handling lions.
Man that was so much more than I initially planned on typing, but trying to get you back on track. Best of luck!
Update in response to OP’s update:
The graph you added is indeed a bivariate analysis but, if I understand you correctly, of the target against a certain feature. This “correlation” study does not have meaning as far as identifying heteroscedasticity in the performance of your model.
Your bivariate analyses should analyze either the target or a single feature against the residuals, or error terms, your model produces. This will show you which features (including the meta feature ‘target above 50’”, are connected to heteroscedastic results and is how you can begin to identify the model’s “blind spot(s)”.
You are almost there!
Example: If you adjust your analysis to be of the residuals, as described, and find that “target > 50” enables a linear separation between two distinct error distributions, then you know you need to either engineer a new feature that your current model can work with, change models to something that can understand that meta feature without additional feature engineering, or train a separate model for this separate process that occurs when t > 50.