0
$\begingroup$

My boss wants me to evaluate a rules based model that a former employee designed. The problem is that I'm not sure what type of model it is or how to evaluate its performance.

The model shows which types of products a certain demographic of customers is likely to purchase.

Here is how the model works: We have customer data consisting of demographic information and products purchased. The model develops a list of rules based on the spurious relationships between the demographic information and products.

For example, suppose 40% of our Chinese, female customers between the ages of 21-32 bought a frozen pizza product. Then the model would create a rule with .4 strength linking Chinese, female, 21-32 age range to the frozen pizza product type.

We then use these rules to decide which marketing materials to send to potential customers. For example, we might send frozen pizza coupons to every female, Chinese, 21-32 year old in the area.

Note that we get the data in this step from a market research vendor, but the model itself was trained on our much smaller, internal customer database.

Does this type of model have a formal name? What's the best way to evaluate this model's performance?

UPDATE: Just to clarify, a rule consists of a set of feature value pairs F, a product type P, and a strength S. F specifies which customers the rule applies to. P is the product type, and S is the proportion of customers in F who bought P. So in the example above:

F={Ethnicity:'Chinese', Gender:'Female', Age:'21-32'} P=Frozen Pizzas S=.4

We use S as a proxy for how likely a potential customer in F is to respond to marketing materials for P.

$\endgroup$
2
  • $\begingroup$ You may need to define some of your terms ('rule', 'strength') and explain your data better to get an answer to this. $\endgroup$ – mkt Sep 11 '17 at 15:33
  • $\begingroup$ check a priori algorithm $\endgroup$ – Laksan Nathan Sep 11 '17 at 15:33
0
$\begingroup$

This sounds to me like Association Rules. As defined by Wikipedia and cited from [0]:

Association rule learning is a rule-based machine learning method for discovering interesting relations between variables in large databases. It is intended to identify strong rules discovered in databases using some measures of interestingness.

The "strength" that has been calculated for the rule is equivalent to the concept of confidence. Other measures of interestingness for an association rule {X}⇒{Y} include:

  • Support [1], a measure of how frequently the itemset X appears in the dataset as the antecedent/left-hand-side;
  • Confidence [1], a measure of how frequently the association rule {X}⇒{Y} is true in the dataset;
  • Lift [1], a comparison of the observed support to what would be expected if X and Y were independent;
  • all-confidence, coverage, entropy, interest, leverage, etc. [2, 3].

Sources

[0] Piatetsky-Shapiro, Gregory (1991), Discovery, analysis, and presentation of strong rules, in Piatetsky-Shapiro, Gregory; and Frawley, William J.; eds., Knowledge Discovery in Databases, AAAI/MIT Press, Cambridge, MA.

[1] Useful Concepts, Association Rules, Wikipedia

[2] Tan, Pang-Ning; Kumar, Vipin; and Srivastava, Jaideep; Selecting the right objective measure for association analysis, Information Systems, 29(4):293-313, 2004 DOI:10.1016/S0306-4379(03)00072-3

[3] Michael Hahsler (2015). A Probabilistic Comparison of Commonly Used Interest Measures for Association Rules. Link

$\endgroup$
3
  • $\begingroup$ Are association rules applicable when the input is different class than the output? The input here is demographic attributes while the output is product types. Don't input and output need to belong to same item set? $\endgroup$ – Jack Sep 11 '17 at 18:17
  • $\begingroup$ @Jack An algorithm like Apriori will be blind to "input" vs "output" and will just see the attributes associated with a given transaction. This can still generate interesting and actionable rules, though, so I don't think that's a problem. You might find rules such as ${Age:20-30, Frozen Pizza} \Rightarrow {Pop}$ (20-30 year olds who buy frozen pizza are disproportionately likely to buy pop as well) or ${Diapers, Baby Formula} \Rightarrow {Age:30-40, Female}$ (diapers and baby formula are generally bought by women between the ages of 30 and 40. $\endgroup$ – user77876 Sep 12 '17 at 12:56
  • $\begingroup$ @Jack (continued) Another approach you could take would be to segment your data based on the input variables you are interested in and discover Association Rules in those specific segments. $\endgroup$ – user77876 Sep 12 '17 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.