Chicken and egg problem in determining the target feature

Question

I have a large dataset of customers in an online shop, where many features regarding their history of buying behavior are recorded: who bought what, at which time, how often they entered the shop, at which points inside the shop the stood longest etc. To some customers I'm sending randomly ads at periodic time intervals, to others not.

Now I want to use this data to predict which customers will react to ads I'm sending them periodically by buying more stuff.

I think the only sensible way to treat this problem is as a supervised learning problem. So I need to define a target feature for myself by defining what "react" means. But here's the catch, since one can fall prey to the following logical fallacy: One could think that if I define my target feature in the following way, I will solve my problem:

\begin{cases} 1, & \text{customer bought at least one item from the ad after he saw an ad}\\ 0, & \text{otherwise} \end{cases}

But this is wrong. Because a model using this target feature this will catch customers who reacted to the as well as customers who wanted to buy that product anyway before they received the ad, so there was actually no need to send them one.

A better feature, that only catches those that reacted and not everybody who bought the stuff is:

\begin{cases} 1, & \text{customer bought at least one item *that he'd normally not buy* after he saw an ad }\\ 0, & \text{otherwise} \end{cases}

But in order to be able to define this target feature, I'd need to do some machine learning to assess if and what he had bought if there were no ads. I have no idea how to do that. Maybe anomaly analysis - but what should I then use as features? So when using this target feature, I need to carry out preliminary machine learning tasks, for which I don't know what the best practices are.

Hence the "chicken and egg" title: Using supervised learning to make sense of the customers depends for the latter feature on some other, to me unknown, machine learning approach - that also needs to make sense (but in a different way) of how the customers behave.

Therefore my final questions are:
1) What would a good definition of a target feature be?
2) In case I settle on my second propose feature, what machine learning methods help me to predict what a customer would normally buy?

Answer 1

I want to use this data to predict which customers will react to ads I'm sending them periodically by buying more stuff. ... Yes, the selection was random. Not all were sent adverts because there was the worry that we might annoy them.

It sounds like you are interested in measuring the causal effect of sending a potential customer an ad (i.e., whether it changes their buying pattern), and you have sent these ads "at random" in some sense that is not fully described. It sounds like you are conducting a randomised controlled trial (RCT) and you are going to infer the causal effect of your control variable. For this kind of project, you should read up on causal analysis and experimental design, and read about confounding variables and mediator variables in causal analysis.

If you have sent the ads "at random" (in a sense that is independent of the other variables) then this severs any statistical association between the indicator variable for sending the advertisement and any variables formed prior to the sending of the advertisement. This means that you remove any confounding variables that would occur prior to sending the advertisement. For variables occurring after sending the ads (e.g., how often the customer enters the shop) these may be affected by the sending of the ad, and so they may be confounders or mediator variables.

A reasonable way to try to infer causality in this type of case is to ensure that you have a properly constructed RCT, and restrict analysis to covariates that are causally determined prior to the sending of the ads. This will ensure that there are no confounding variables, and it should allow you to make causal inferences via standard regression/prediction methods.

Answer 2

customer who bought at least one item that he'd normally not buy after he saw an ad

I'd use some kind of recommendation system to get this information.

You could use collaborative filtering (here a non specific guide) on all your customers, in order to find similarities between users (two users are similar if they buy the same stuff).

For each user you will have a list of items that have been bought by other similar to him (with some degree).

Then for each user that has seen an ad, you could track if he has bought something which is not in his list (list of possible items). If it's not in the list and it's an item of the ad, then response = 1.

Useful links and examples:

Link 1 Link 2

Resources to get started with collaborative filtering:

CF Link 1

CF Link 2

Edit:

To answer your question, it'd be really difficult to prove causality, with a simple classification problem.

For starters, I'd use something along A/B testing.

For each customer, you use CF in order to get clusters of similar habits. Then feed the ads to half of the cluster.

You could then check the behaviour against the two groups, did the purchases improved for the ad-recevied group? Or is it the same?

This should help you shed some light on the causality of the ads.

Answer 3

I take it you are doing this through an A/B test? If so there is are a few methods you could implement to alleviate the apparent paradox. You could synthesis a composite series through contrasting the behavior of two similar products. Determining what is a similar product is an expansive topic in itself, but for the sake of this explanation lets assume you have a similar product. You compare the sales of each. Use the similar product as a benchmark for base state sales. Use the benchmark to strip out the sales that would have occurred base state in the target product. As a result, you are left with an approximation of the sales that are generated from the advertising. This is a high level approach to solving the question if ads are driving more sales.

You can then take this a level deeper to approximate customer behavior. Now you have an rough number of those who purchased the product because of the ad. You just have to determine which customers are have the highest probability of purchase because of the ad. Examine the customer features ( click patterns, time spent, frequency of visits and past purchasing behavior ) to identify which of those customers bought because of the ad. A really simple identifier is where you delivered the ad. Was it delivered on the homepage, another product groups page or the target product's page. Compare this with the customer's normal click patterns and bingo you might see they deviated because of the ad. Using these factors you should be able to match the customers to the group of who bought because of the ad.

The answer above is a good approach too.

Update

For A/B test bit, I was referring to directing a portion of the customers to the ads. I guess it doesn't matter too much.

What I am proposing for the composite series ( maybe better labeled composite data ) is to remove the amount of sales that would normally be made in any given period to isolate the sales driven by the ad.

As you can't use historical data for this test and you are changing the real-time data by introducing a new variable, you must find a proxy for what this normal sales level would be in real-time/concurrently. You can use single/multiple similar products to calculate what the base level of sales is. Take this level extrapolate it to the target product. Subtract extrapolated value from observed target sales and that will give you an approximated value for target product sales generated from your ad test. The idea is to find the level of sales that would have occurred in the target product had you not run the test.

This will solve the question of "what number of customers purchased because of the ad?". Now you just have to examine the customers you sent the ad to and determine which have highest probability to have purchased because of the ad. Then you can get to work delving into the behavior specifics. The goal of this is to narrow down the amount of customers you have to analyze while giving a high level view of the ad's performance.

As a note, determining what is a similar product is a large task. You can use correlations along with historical sales data to determine a good comparable.

Please let me know if that clarifies things.

Answer 4

First this is most basic problem in Marketing Analytics, may be some books on marketing analytics would help and your question is too vague and comprises of several questions in one but still i am trying please comment if i reach there.

where many features regarding their history of buying behavior are recorded: who bought what, at which time, how often they entered the shop, at which points inside the shop the stood longest etc. To some customers I'm sending randomly ads at periodic time intervals, to others not.

I have been using gains and lift chart for assessing what better you can get when you send targeted marketing campaigns or random. Here is an amazing link that might excite you.

But in order to be able to define this target feature, I'd need to do some machine learning to assess if and what he had bought if there were no ads

For this problem use Marketing Mix Model.