How to identify which features are more likely to contribute to the desired outcome?

Question

This question is in tandem with my earlier question here:
https://stats.stackexchange.com/questions/188799/using-ml-approaches-to-build-a-recommender-engine-for-sales-team

However, now I'd like to discover insights about x features as opposed to predicting future y values. To clarify, here's a bit of context: I'm a developer at a startup, where most of our revenue comes from ad purchases. We have a sales team that approaches our leads to sell them ads. I wanted to make this process more efficient by looking at past data (particularly those leads that converted) and discovering patterns about their featues. That is, if we find out that leads with certain features (x^k or x^j) are more likely to convert, the sales team would spend more time and focus on those leads.

So, to formulate the problem, I guess we can say, we have the following data from past sales efforts: {(x₁, y₁), …, (x_n, y_n)}, where x represents the sales leads and y is from a binary set of {0,1}, where 1 indicates that the lead converted and 0 denotes that she didn’t. And each sales lead has the following feature vector for x_i in R^d: x_i = {x¹, ... ,x^d}

I believe we'll only need to focus on the Xs that did convert, right?

How would you go about solving this problem?

Answer 1

First things first - this is not a feature selection problem. Not yet, anyway. It is the definition of a prediction problem. So let me get you started with that --

You have a bunch of advertisers, to whom you want to sell ads. You know their features. You also have a bunch of advertisers you've pitched to in the past - some were convinced, others weren't. Now, you want to predict, given your past sales records, which advertisers are most likely to buy ads from you. Right?

The way to do this is to build a model, that relates features, to outcomes. In this case, with a binary outcome, you could use logistic regression. Once you've fit that model (using ALL your data, or else you'll think everybody will always buy from you), you put the new advertiser's features into that model, to predict how likely they are to buy, also - or, more accurately, how much they resemble the advertisers who bought from you in the past.

Maybe once you've done this you'll find that some of your features don't help the prediction, or there are too many, in which case you can remove them from the model. But until you've built the model (which is more work than it sounds, done right), there's no reason to throw away any of your features - you haven't used them yet!

I'm sure you'll have more questions once you get started, but the first thing for you to do is to start reading about prediction - that's what you're trying to do here.

Answer 2

related: What Machine Learning Algorithms are Good for Estimating Which Features are more Important?

The problem you posed can be described in a few different ways: I would call this a feature selection or variable importance problem. Alternatively (and perhaps more appropriately, as described in Sophologist's answer) we can constrain this as a prediction problem. Either way, the short answer is that there are multiple different ways to get the importance of different predictors, and that more importantly there are two bigger questions here: how much does a variable (or set of variables) contribute to a particular model's performance, and how relevant is a variable in general? The second question is tricky and I'm not sure there's an easy answer to that. So that being said, and taking inspiration from Sophologist's answer, it would be most straightforward to build models with your feature vectors to figure out how the different features interact with each other to affect the predictions of those models.

Without knowing more about the type of features in your data, what your data looks like, and any other goals you might have with the data, it's tough to suggest any one method over another. Like above, and like others have mentioned, a relatively intuitive thing to start with would be to find how all the features interact perform for some model (say, logistic regression), and then inspect the subsets of the features in subsequent iterations. There are a variety of heuristics that you can do to search the subset space for the best subsets. There's some info on the wiki page here. It may actually be the case that removing "less important" features hinders the overall performance of your model! See (this question)[Should covariates that are not statistically significant be 'kept in' when creating a model? for more clarification on that. As for what part of the data to use, no matter which method you use it would behoove you to consider both the positive and negative data. It should also be stressed that you may not get one universal ranking of variable importance/feature 'strength' across these methods, as many of them are tied to how important that variable is in a specific context (e.g., how it impacts the performance of a random forest). See this blog post for more clarification on these different metrics, and a post here about some of the R feature selection methods with some examples.

Answer 3

After you trained a model over your data, then a simple way to understand which features are more positively or negatively contributing to an output is by looking at their corresponding weights. Generally speaking, the larger the weight for a feature the more positively that feature contributes to the output and vice versa. In linear models this is 100% true but in non-linear models it is harder to tell. For instance, in a logistic regression you can simply identify the important features by looking at the weights.

Answer 4

Though the post is almost 2 years old now, it is worth mentioning that OP's question falls into general trap of survivorship bias.

If you try to find common values / levels of features for positive outcomes (conversions) only, you implicitly state that bad outcomes have very different set of characteristics / values - is this true? Maybe, but most likely not.

E.g. you take only the leads that converted to sales and notice that all those clients a located in same area. This might lead to false conclusion that a) this particular area is worth spending more sales effort and b) there are no non-converted leads in this area; clearly that can't taken as ground truth unless you also checked the rest of leads for that area. You might find equal (or larger) number of non-conversions.

In general features should come first, which brings us to usual prediction problem. That was already answered above.