Data scientist interview question: Linear regression low $R^2$ and what would you do

Question

I faced an interview question for a job where interviewer asked me suppose your $R^2$ is very low (between 5 to 10%) for a price elasticity model. How would you solve this question?

I couldn't think of anything else other than the fact that i will do regression diagnostics to see what went wrong or if any non linear method should be applied. Somehow i think interviewer was not satisfied with my answer. Is there something else that is done in such a scenario to fit a model and use it for production level prediction despite it having low $R^2$?

Edit: At a later stage they gave me the data to model the problem during interview and i tried adding lagged variables, impact of competitor price, seasonality dummies to see if it made any difference. $R^2$ went to 17.6 percent and its performance on holdout sample was poor. Personally i think its unethical to put such a model for prediction in live environment as it will give erroneous results and result in clients loss(imagine using pricing recommendation from such a model on your company revenue!). Is there anything else that is done in such scenarios which is too obvious that everyone needs to know? Something that i am not aware of, which i am tempted to say 'a silver bullet'?

Also, lets imagine after adding exogenous variable $R^2$ improves by further 2% then what can be done in this scenario? Should we discard the modelling project or there is still some hope of developing a model of production level quality which is indicated by performance on holdout sample?

Edit2: I have posted this question in economics.stackexchange.com forum for understanding this problem from the perspective of economics

Answer 1

What if we look at the problem from this perspective. Price elasticity is the relationship between demand and the price of a product.

When r-square in this situation is low, we could then possibly imply that the relationship between price and demand for that particular product is not a strong one.

From a pricing stand-point it could mean you've found a product for which you can price arbitrarily without a large impact on demand OR that demand is quite erratic despite differential pricing.

If you look at Veblen goods, they are examples where elasticity is inverse. As price increases, demand increases.

If on the other hand, r-square is low, it could simply mean a category of product for which the price is relatively unimportant when it comes to demand. Of the top of my head, a cancer drug could be something that could adhere to this property. Where the importance of the drug outweighs the price it commands and could show no change in demand.

And in conclusion, I'm assuming the intent of the interviewer might have been to judge if you knew what the implication of a low r-square meant instead of finding out how to build a better model with a higher r-square.

Answer 2

I'm not sure what the interviewer was after but when facing a poorly preforming model these are the things I consider and an answer I would love hearing as an interviewer (been interviewing for a couple of years now).

1. Getting more data: This might not always help but there are few things that can help you evaluate this solution effects:

   • Run the model with different sample sizes - if results improve with more data then its reasonable assuming getting more data will continue improving model performance.
   • Features to sample ratio - after you selected features try understanding if you have enough samples per each feature value. See an answered question on this subject.
   • Missing target values - elasticity might not behave similarly between different price ranges. In a situation where you samples data is biased towards a specific range there is a good chance that you won't be able to generalize(for example 90% of samples are for prices between 0-10 and the other 10% are for prices between 1000-10000). There are ways tackling this problem other than getting more data(split the model training, don't use regression).

2. Better feature engineering: If you have enough data and you know about deep-learning then maybe this one is irrelevant. In case you don't fit the mentioned criteria, focus your efforts on this one. In user-behavior models, there are many relations that our human-intuition is better understanding than a machine trained model.
   As in your case where you engineered a couple of more features and improved model performance so greatly. This step is prone to errors since it usually involve logic based code(If Elses/ Mathematical formulas).

3. Better model selection: As you suggested, maybe a non-linear model will work better. Is your data homogeneous? Do you have reasons to believe that cross features will explain the price elasticity better? (seasonality * competitor's price).

4. Hyper parameters tuning: grid searching model's hyper parameters(+ cross validating results) is a good practice but as far as to my experience it rarely improves performance greatly (surely not from 5% to 90%).

There are more things that can be done, but these points are generic enough.

Answer 3

On top of what suggested by @DaFanat and @Arun, I would like to add that some visual inspection might help.

For example, it might be the case that some outliers impact your $R^2$. Having worked on revenue management problems, I had to constantly investigate influential points. Very often outliers were associated to specific one-off events such as promotional campaigns, discounts, etc.