2
$\begingroup$

I'm working on clustering email addresses using K-means based on their value to and engagement with the company (metrics such as % of emails opened, # of web browsing sessions, etc). I would like to use days since last purchase as a feature to model on (capped at 365 days), but ~37% of the email addresses have never made purchases.

What would be the best way to deal with these missing values? I don't want to remove those data points as they are still important and make up a significant portion of the entire data set.

Note that I have data on email subscription date - could this be used in place of the missing values?

$\endgroup$
0
$\begingroup$

Missing data is always tough to deal with, and there is no definitive right answer. There are multiple ways to perform data imputation, from nearest-neighbors (take the average of the closest data points with non-null value in the relevant field), to global max/average. Each of these have application-specific pros and cons, and I'd try to look at your data or experiment to see which one of these methods is most appropriate.

It is also possible to use the email subscription date and simply copy/replace that feature, but again, it depends on the application, and I don't think there's a single right answer.

As an added point, I would also ask yourself if k-means is really the best way to cluster this data given the features that you have. A good way to validate this assumption is to see if the similarity metric of k-means (intra-cluster variance) would result in a "good" cluster for your data, especially since you have missing values. In other words, think about what points "should" be close together and what points "shouldn't" be close, and work backwards from there to find an appropriate algorithm. For example, here's a decent survey of a few other clustering algorithms. K-means, however, is a great starting point.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I suggest one of the two possibilities:

1)Instead of using last purchase as a feature, you could use (last purchase)/(time in the system) or 1-(last purchase)/(time in the system).

Note that in this approach 1-(last purchase)/(time in the system). The subjacent hypothesis is that a customer that has this variable close to zero is very similar to the guy that is in the system and never bought. Note that this might be true.

2) Besides the variable last purchase, add a dummy variable that indicates if the costumer has already purchased anything.

3) If you have doubts about the hypothesis behind approach 1, you can also add a dummy here as the second option above [i.e., use (1) + (2)].

Anyway, if have doubts what is the best approach for your data, use cross-validation to check which one works better.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Regarding your first suggestion, I would still need to quantify the null / no purchase values. I have played around with using (# of orders) / (time since opted in) as a feature. I have also played with (# of orders) and (time since opted in) as their own features, but am not sure how valid either of those methods are. $\endgroup$ – ERB3 Feb 20 at 0:08
  • $\begingroup$ See the edition. Now you actually 3 options that can be tested using cross-validation. $\endgroup$ – DanielTheRocketMan Feb 20 at 0:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.