How can I quickly detect cheating variables in large data? Suppose We have a data set with millions rows and thousands columns and the task is binary classification. When we run a logistic regression model, the performance a lot better than expected, e.g, almost perfect classification.
We suspect there are some cheating variables in data, how can I quickly detect it? 
Here cheating variables means a variable that is very indicative to the response and we should not use it. For example, we use if a person make customer service call to predict if a person purchased a product or not.
 A: One way of detecting cheating variables is building a tree model and look at first few splits. Here is a simulated example.
cheating_variable=runif(1e3)
x=matrix(runif(1e5),nrow = 1e3)
y=rbinom(1e3,1,cheating_variable)

d=data.frame(x=cbind(x,cheating_variable),y=y)

library(rpart)
library(partykit)
tree_fit=rpart(y~.,d)
plot(as.party(tree_fit))


A: This is sometimes referred to as "Data Leakage." There's a nice paper on this here: 
Leakage in Data Mining:
Formulation, Detection, and Avoidance
The above paper has plenty of amusing (and horrifying) examples of data leakage, for example, a cancer prediction competition where it turned out that patient ID numbers had a near perfect prediction of future cancer, unintentionally because of how groups were formed throughout the study.
I don't think there's a clear cut way of identifying data leakage. The above paper has some suggestions but in general it's very problem specific. As an example, you could definitely look at just the correlations between your features and target. However, sometimes you'll miss things. For example, imagine you're making a spam bot detector for a website like stackexchange, where in addition to collection features like message length, content, etc., you can potentially collect information on whether a message was flagged by another user. However, if you want your bot detector to be as fast as possible, you shouldn't have to rely on user-generated message flags. Naturally, spam bots would accumulate a ton of user-generated message flags, so your classifier might start relying on these flags, and less so on the content of the messages. In this way you should consider removing flags as a feature so that you can tag bots faster than the crowd-sourced user effort, i.e. before a wide audience has been exposed to their messages.
Other times, you'll have a very stupid feature that's causing your detection. There's a nice anecdote here about a story on how the Army tried to make a tank detector, which had near perfect accuracy but, ended up detecting cloudy days instead because all the training images with tanks were taken on a cloudy day, and every training image without tanks was taken on a clear day. A very relevant paper on this is: "Why Should I trust you?": Explaining Predictions of Any Classifier - Ribeiro, et. al. 
