I am in the process of learning machine learning/regression on the job so while I have some basic understanding of the algorithms, I'm a beginner so please go gently on me. For what it's worth, I've read An Introduction to Statistical Learning cover to cover and feel I have a good understanding of the concepts.
I've also read the guidelines and have attempted to make this a really good question. If you have additional suggestions on improving the question, they are very welcome.
My task: understand what makes our customers place their 2nd order. So this is a classification problem where the response is "placed 2nd order within 6 months of being acquired"/"not placed 2nd order within 6 months of being acquired".
My dataset is 1 customer_id per row. Each customer has a number of features. Dataset has 150k observations. So far I've identified ~100 potentially interesting features based on prior experience, prior analyses, gut feeling etc.
I am at exploratory data analysis stage where I am trying to get very familiar with the data and potentially weed out useless features.
My current issue: Based on empirical knowledge we know that return rate on 1st order likely has an impact on whether the customer will/will not place another order.
I have 5 features that all reflect return behaviour in one way or another:
Customer has/has not returned something from the first order (binary, 1=has returned at least 1 product, 0=has returned nothing)
Customer has/has not returned everything in the 1st order (binary, 1=everything was returned, 0=nothing or something was returned)
Return rate for first order (expressed as a continuous value between 0=nothing returned and 1=everything returned). Calculated as units returned/units ordered. NOTE: the majority of our orders only have 1-2 units so this mostly is either 0, 0.5 or 1.
Return rate for first order but only when something was returned (same as 3 but missing 0 boundary. The reasoning is that this feature and 1 are complementary and together they cover the full behaviour).
Number of units returned.
Below are the histograms of the 5 features, labelled 1-5 (bigger if you open image in new tab):
NOTE: "Units returned" is highly skewed and follows a very similar pattern to "Units ordered".
And point plots showing the mean of each feature for the two values of the response "placed 2nd order" / "did not place 2nd order" (bigger if you open image in new tab):
The pointplots seem to support the theory that 100% return behaviour in 1st order has a big impact on whether the customer places the 2nd order and generally that a high return rate is likely influential whereas "some" return behaviour may not be influential. But it feels like all 5 metrics are severely impacted by the distribution of behaviour reflected in the histograms. It makes it difficult for me to work out which one is the most relevant. I am veering towards feature 3 (return rate) as that truly captures the entire range of behaviour.
My goal is prediction but ALSO greater understanding of customer behaviour (I know that inference/predictive power are often distinct goals. Here, I would be willing to sacrifice predictive power in order to get a greater understanding of customer behaviour)
My questions are:
1. Is there a general structured process/guidelines for selecting from among features which essentially express the same behaviour before building the model?
Why before? From my understanding, chucking everything into a model is not advisable. Here, there are only 5 features but I have a large number of behaviours which could all be expressed in a similar fashion (e.g. cancellations, bought product category X, etc). I could easily end up with 200 or so features. Surely I should try to avoid that?
2. I will try LASSO and/or Ridge regression as potential algos on my problem. Should I even worry about narrowing down the feature space beforehand or can these algorithms make sense of very similar features?
3. Should I treat these 5 features as a group and include/exclude all of them from the model at the same time?
4. How does feature selection in this case fit in with the cross-validation process?
I've read several answers/comments on here that recommend that feature selection be done separately, on each fold, rather than beforehand. That makes sense to me but it was not clear to me whether feature selection on each fold is done:
a. automatically by virtue of the algorithm selected OR
b. whether there's an element of the analyst's own judgement going into the process (EDA being done on each fold).
If the latter, then it feels like the whole process of "feature selection" quickly becomes unmanageable.
5. Any other suggestions on how to approach feature selection in this particular case?