I am looking for a book/case study etc on how to build a fraud detection model at the transaction level. Something applied rather than theoretical would be really helpful.
Fraud detection is a rare class problem. Chapter Six of Charles Elkan's Notes for his Graduate Course in Data Mining and Predictive Analytics at UCSD walks you through the prediction of a rare class, and the pitfalls and proper ways to evaluate the success of such a model. The methods he specifically uses are Isotonic and Univariate Logistic Regression. The software he uses in the class is Rapidminer, but I prefer R. If you choose to use R, you can perform both of these functions using the isoreg and glm functions. Many people also like to use SVMs in fraud detection, but part of the model selection criterion should be the speed with which you need to validate the transactions. If, for example, it's the swipe of a credit card, SVMs are wholly unfeasible because it will take far too long to process. This is why, in production environments, variants on regression models are typically used for fraud detection.