Getting hints from target variable by only looking at the extremes: Will it ruin predictive power? I have a large set of predictors and a target variable which is extremely difficult to model. After a couple of failed trials (glm, DT, RF, NN) I got the impression that it is almost random noise.  
Recently I tried a trick which from theoretical point of view is a "No-Zone": I chose a relatively small subset of the data where the target variable is either larger than a positive threshold or smaller than a negative threshold. Basically I chose the extreme points. Then I tried modeling, and I got significant improvements. 
I know that it practically yields to no predictive power because we don't have a priori knowledge whether a point is extreme or not. But the fact that it could be modeled at extremes, means that there is valuable information there and it is not pure noise. If it was pure randomness, then I shouldn't have got correct predictions at the extremes. I am attaching a side by side plot here. The occurrence of extreme points is about 1:15 in the original dataset. The scatter plot on the left is the prediction of model for extreme points and the one of the right is the same model when applied to common points (non-extreme). The contours show point density. 

My question is: Am I right to say that due to success at extremes there is valuable information that can in someway be extracted here? And if yes, then how to extract it and use it "predictively"? 
 A: Counterexample: imagine you want to learn to predict human income given some other data. You train your model on the data on billionaires vs people who live for \$1 per day. Those two populations do not have much in common with vast majority of general population. Additionally, if you're using something like squared errors (including $R^2$), correctly predicting very high values potentially saves you from making large errors (squared!), what can have significant, while misleading, impact on your error metric. So this seems to be a risky, and problematic, strategy.
A: Putting the statistical relevance of a model created from a (non-random) subset of your data aside for a moment, judging from your first plot I don't think that you are actually getting predictivity from your second model, either. 
It looks to me like your data is a large, noisy blob, centred near the origin -- when you cut the centre of the blob out of the picture by discarding the low target values, you get two noisy blobs, which happen to be centered s.t. the predicted values are near the target values.
To answer your question, leakage of information from the target variable will not necessarily ruin predictive power completely, but in general should reduce it or overstate the true strength of the relationship. In this case, unfortunately, it doesn't look like there is much predictivity to ruin. Hard to tell from those two graphs, of course, but it sounds like you have looked pretty hard for a relationship here. Is there a strong theoretical reason to believe that there is a non-random relationship to predict?
