Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have developed a recommendation system which recommends the products based on the transaction history of the customer. All I have is one year of transaction data with no information on purchase reasons like due to recommendation system or email campaign etc.
I need to validate my results before pushing it to production. Is there any method to do this?
Thanks
Unfortunately, you may not be able to validate your model correctly at all before going into production without having additional data. It sounds like you used all the available data to build the model for your recommendation system. This is not the way to approach the predictive model building process. For future reference, at the very least what you should have done is split your data into two parts:
To be very thorough you could have split your data into 3 chunks and reserved your last chunk of data for validation of your final model prior to deployment to production.
Why not validate against the data you have?
You might ask yourself why you can't simply validate the data with the data you used to build the model? Well, that's because you may have "overfit" your model to the particular features of the dataset--and not necessarily to the general features that drive purchases. So, had you set aside 20% of your data before you built your model, then once you found a few candidate models for your recommendation system, then you could have validated each one of those models against your "hold-out" or validation dataset. You'd then want to select the one that had the least error, or the one that led to the greatest proportion of recommended items that were purchased (or greatest recommended profit, etc.).
What to do now?
Given that you've already built the recommendation system with all your data and you no longer have any data for validation, I believe your only choice now is to deploy your solution to production, but randomly select customers that will receive recommendations from your system. You will then have some data that you can use to compare profits, purchases, etc. from those who received recommendations from your model versus those who did not receive recommendations from your model. There are a few different sampling schemes you can use for this purpose, so you should research those. You might also want to review articles like this to determine how you might want to go about validating in production.
Best of luck with your new system!
Check how well your clusters (bad vs. good recommendations) are separated. Various distance metrics are available for such comparisons. See also my article on the elbow rule at https://dsc.news/2EYkh3E. The strength of the signal is an indicator of how well your classes are separated. Try different models, select the one with the best discriminating power.
