# How to do multivariate machine learning? (predicting multiple dependent variables)

I am looking to predict groups of items that someone will purchase... i.e., I have multiple, colinear dependent variables.

Rather than building 7 or so independent models to predict the probability of someone buying each of the 7 items, and then combining the results, what methods should I look into to have one model that accounts for the relationships between the 7 related, dependent variables (things they can purchase).

I am using R as a programming language, so any R specific advice is appreciated.

Based on your description, it appears a multinomial logistic regression is appropriate. Assuming your outcome is a factor with 7 levels (one of the 7 buying options), then you can quickly predict membership using a multinomial logistic regression model (see ?multinom in the nnet package in R). If your outcome cannot be combined into a factor with 7 levels, then a cluster analysis would be needed to group the items together before fitting the multinomial logistic regression.

• It is not multinomial regression. I have 7 different products, each product has up to 4 factors.... there are strawberries, and types of strawberries, and then milk and different types of milk, and apples and different types of apples, and I need to predict the correct shopping cart... so green apples with farmed strawberries, with 2% milk etc., – blast00 Apr 20 '14 at 22:26
• I have your solution! I'd recommend polytomous latent class analysis, in which the outcome is a set of factors that are assumed to group in one or more latent classes. Membership in these classes is predicted based on a multinomial logistic regression. See ?poLCA in R for more information on fitting this model. – statsRus Apr 20 '14 at 22:30
• I am reading through this - thank you statsRus. There must be other ways though. – blast00 Apr 20 '14 at 22:49
• Specifically, machine learning methods, since I do not need to fit a probability distribution / am OK with a black box model – blast00 Apr 20 '14 at 23:07
• Keep in mind a great deal of statistical models are in fact unsupervised machine learning models -- but you're right we usually care about the inputs with these models. For supervised machine learning with many inputs and outcomes (and a black box quality), I'd suggest neural networks (?nnet in R). – statsRus Apr 20 '14 at 23:11

You could build a random forest where each of your classes is a group of items (i.e. "green apples with farmed strawberries, with 2% milk"). Then, based on the characteristics of the shopper or whatever your predictors are, you can provide a predicted probability of purchase for each group of items. I would use R's randomForest package (https://cran.r-project.org/web/packages/randomForest/index.html) to do this.

One option is to obtain frequencies of all the combinations of product purchases; select the few most common combinations; then build a regression model to predict each individual's chosen combination. E.g., with a binary logistic regression you could conceivably predict purchase of a) White Wine, Brie, Strawberries and Grapes vs. b) Red Wine, Cheddar and Gouda. With more than 2 such combinations, or if you want to include the category of "none of the above," multinomial logistic regression would probably be the method of choice.

Note that including just the common combos means you will have more workable numbers of each but that you will be excluding the others, at least from this procedure. I could imagine 7 items creating dozens of combos each chosen by at least a few people. This is possibly too many categories for your sample size. Moreover, if a combo were chosen by just a few people, your model would have very little information to work with.

Another option is to use cluster analysis to arrive at a few sets of items that tend to be purchased together. With 7 items, you'll probably end up with fewer than 4 clusters, which might make your task easier. If you try cluster analysis and find the results unworkable, there is no reason why you have to use them: just go back to the frequency-based approach described above. In this case, if I read you right, you're looking for the most descriptive and interesting array of categories, and in establishing that, you don't need to worry about degrees of freedom or multiple comparisons or any such concerns that might apply if you were trying out multiple methods in performing some inferential test.

• Thanks for this suggestion. There must be multivariate machine learning methods though. Simliar to how you might have 2 dependent variables in an "easier" regression model.. and you just do lm(y+z~...) .. I think.. – blast00 Apr 20 '14 at 22:47

I am assuming that you want to analyze situation similar to the following;

Yi = f(X), where f() is a non-linear link and X is a vector of covariates and Yi is the i-th dependent variable, which is ordinal in nature (if it is categorical Yi cant have more than two categories), and say in your model i = 1, 2, ...5 and each of the Yi s is correlated... If so, you can certainly employ Multivariate Probit. R, Mplus and SAS can estimate MVP

In contrast, you have Y = f(X), and Y (notice there is only one Y) is categorical and for example, has N categories so that that choices made over the N categories are exclusive and exhaustive; you need to fit Multinomial Logit model. There is something called multinomial probit as well, simialr to multinomial Logit.

Hope this helps. Thanks Sanjoy