0
$\begingroup$

Recently I'm helping a company building a model predicting if the web visitor is a potential high value customer based on visitor's web behaviors.

To make the model easy to understand, I suggested to start from a decision tree model. However, the technical leader in the company asked one question: Why not just find past visitors in our database that have exactly same attributes with the new visitor and get the probability we want by calculating the high-value probability of past visitors?

I have thought about the answer for quite a bit and the only reason in my mind is that we may cannot find a exact match in our database.

Could someone give me some idea? Thank you in advance!

$\endgroup$
0
$\begingroup$

Let's call using a decision tree (or most any other statistical model) A and the method the "technical leader" suggested B. To be precise, what B would have you do is find the proportion of users in the training data with identical features (attributes) who turned out to be high-value, and predict that as the probability of the new user being high-value. You'd like to know the true probability, but of course, you have only a finite sample of users, so all you have are sample proportions, which are estimates of true probabilities.

The chief substantive difference between A and B is what kind of data is considered when making each prediction. B considers only subjects who had identical features to the subject you want to predict something about. A considers all subjects in the training data, trying to infer the separate effects of the various features.

One consequence of this is that, as you noted, B can't make a prediction when a subject has a feature vector that isn't in the training data. And when the training data has only a few subjects with the given feature vector, B will in general make very liberal predictions that are likely to overfit. For example, if you have only one subject with the given feature vector, and they turned out to be high-value, B will predict that the new subject has a 100% chance of being high-value, which is obviously unrealistic.

In practice, B will perform as well as A only when there's a lot of training data available for every possible feature vector. This is generally only the case when there's a small number of features (say, 3 or less), each of which has only a few possible values.

Of course, if you or the technical leader are skeptical, you can estimate the predictive accuracy of both methods with the data you already have, using cross-validation or a similar technique.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.