I have a binary classification problem where I'm required to classify transactions as anomalous or normal (1 or 0 respectively), with anomalies being the rarer instance.

With what I know to be true from experience about the domain & the ecosystem where the transactions happen, I chose 4 variables out of many and ran a logistic regression. Sure enough, all 4 variables were 'very significant'.

My question is, did I just indulge in data dredging (or any other less-than-ideal practice) by cherry-picking my variables before-hand without checking if they are statistically significant independently, or is this a benign case of domain knowledge in action?

EDIT: I wanted to tag data-dredging, but couldn't.

  • 1
    $\begingroup$ If you haven't acquired this pre-knowledge from the same data set you ran the regression, it is not data-dredging, it is domain knowledge. $\endgroup$ – Cagdas Ozgenc Aug 9 '17 at 13:16
  • $\begingroup$ No, said knowledge was acquired over time observing trends over time on a large number of transactions. Thanks @CagdasOzgenc $\endgroup$ – H-Finch Aug 10 '17 at 4:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.