Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
If you read my question you will understand my concern regarding application of poisson distribution and therefore decision making based on poisson probabilities may be not the right thing to do. So question now becomes how to fit the right distribution and how much to trust the probabilities for taking business decisions
Thanks Anon, example is very practical but assumption that addimissons follow poisson may be very simplistic. I had a doubt on application of poisson that it gives very low probability for (calculated using survival function) for higher values of the data. I posted it here datascience.stackexchange.com/questions/61485/…. So the issue is in books we are taught that for admissions or bus arrivals or restaurant orders follow poisson but there are practical issues in Application to real problems
by above comment all I mean to ask is : In tree based models since we full encode a categorical feature I mean we don't drop a level as reference, can we drop unimportant dummy features and keep only important dummies?
@ldin If you drop some dummy variables then coefficients of remaining dummy variables would not be accurate and no inferences about features importance could be done or ranking of features to select the most important ones based on coefficients would not be possible. That's my concern –
Thanks Kjetil for your inputs, a question: in your answer "Does it make sense to apply recursive feature elimination on one-hot encoded features?" you suggest don't drop dummy features of a particular categorical variable if you have dropped a level/category while encoding. I ask if I fully encode a categorical variable i.e. I don't drop a level, then will it make sense to drop some levels and keep important ones only since there won't be any dependcy in the sense that there isn't any dropped level or category. I guess in tree based models dropping unimportant dummy variables is fine?
@ttnphns "Dropping just one dummy of the set is essentially merging of this category with the reference category">>>>I agree, so to overcome this merging of dropped dummy features wouldn't it be better to use full encoding i.e. not dropping a level of categorical variable as reference category. If that's done then even if you drop 1 or more dummy features it would not create an issue as all dummy variables derived from a particular categorical variable would be independent in the sense that there is no reference cateory
