3
$\begingroup$

I am currently trying to build a stacked algorithm in order to determine how many people in each region of a country will be likely to buy a product versus its competitors. I have some data from an online survey which collects user's demographics as well as their preferred product and I've weighted these so they reflect my customer base correctly.

I currently have three models that I am happy with (XGBoost, Random Forest and Multi-Layer Perceptron) - they give me probabilities that I can then use to poststratify to determine how many people in each region will buy each product.

I now want to stack these models together so I can get a better model that incorporates all of the predictions. I would like to do this so that at the end I get some distributions of how many people I might be able to target in each region, rather than just a point estimate. I have tried building a model in BRMS (R) which uses the predicted probabilities from each model and treats them linearly, e.g.

product ~ prob_A_XGB + prob_B_XGB + prob_A_RF + prob_B_RF + ...

where I have dropped one of my products to avoid multicollinearity in the model. Unfortunately, my meta model gives very different predictions to my base models. I have also tried building a model which uses a majority voting by using the predicted product from each model but this performs even worse.

Is there anyway to essentially take a "weighted" average of the class probabilities in my base models that will give me a distribution of the potential number of customers in each region?

$\endgroup$
3
  • $\begingroup$ I have read your post twice, and am still confused. On the one hand, you seem to want a prediction of the number of people that buy your product. But on the other hand, you write about predicted probabilities. And you seem to be switching back and forth between the two... can you please clarify? $\endgroup$ Commented Mar 1 at 17:25
  • $\begingroup$ I have a post stratification data frame which is in a long format which groups people together based on their age group , gender, income group and region. Each of these groups has a number of people in it. I have ran classifiers using my survey data to predict which product someone will buy based on their demogs (and info about their region). These models give me probabilities of buying a product based on demographics and regional info. I then run my post strat data frame through my models and use the predicted probs to estimate the number of people who will buy each product in each region. $\endgroup$ Commented Mar 1 at 17:30
  • $\begingroup$ OK, thanks, that helps. $\endgroup$ Commented Mar 1 at 17:33

1 Answer 1

2
$\begingroup$

The simplest possibility would be to just take unweighted averages of your predicted probabilities. This corresponds to equal weights, and will give you bona fide predicted probabilities.

Alternatively, you can of course use any kind of weighting in a weighted average. For instance, you could give higher weights to models that performed better in the past. (Ideally, on holdout data, not in-sample - in-sample performance is prone to overfitting.) You could map any performance measure to weights in a way that the weights sum to one and such that a better-performing model gets a higher weight. How exactly you would do this would depend on how better performance is operationalized in the first place. For instance, "smaller is better" AIC values can be turned into "Akaike weights".

However, do also try unweighted averaging. For one, it's easier. Also, unweighted averaging of predictions is quite often better than trying to find "optimal" weights for weighted combinations. This has been called the "forecast combination puzzle", and Claeskens et al. (2016) give an intuitively appealing explanation: estimating weights comes with uncertainty, and this uncertainty directly degrades the accuracy of the combination predictions.

In addition, while this is a perfectly feasible approach to your final objective - predicting a total number of customers -, I would also recommend you try other methods, like predicting this number directly. Once you have both methods up and running, see whether taking the average of the two predictions yields better results than each one separately. Averaging predictions very often outperforms separate models, or trying to pick a "best" model. (Which is actually just taking your approach one step further.)

$\endgroup$
4
  • $\begingroup$ Thanks! So far I've done unweighted and used a random forest meta model to extract importance and use that to weight my averages. I am stuck on how to then go about generate distributions (even if only of probability). I have thought about using something like BRMS then extracting the coefficients and using these to infer weights but this feels too complicated. Have you got any advice on how to extract some distributions of plausible probabilities (Bayesian analysis)? $\endgroup$ Commented Mar 1 at 17:49
  • $\begingroup$ What kind of distributions are you looking for? $\endgroup$ Commented Mar 1 at 17:52
  • $\begingroup$ Each group has a predicted probability of buying product A, B, and C. But these predicted probabilities are just the most likely probability extracted from some underlying distribution. I want to sample these multivariate distributions to get the "posterior" distribution of these probabilities. $\endgroup$ Commented Mar 1 at 17:56
  • $\begingroup$ If you have a group of size $n$, and each member has a probability $p$ of buying one unit, and if purchases are independent, the total number of purchases is binomially distributed with parameters $n$ and $p$. $\endgroup$ Commented Mar 2 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.