Timeline for Is it a good practice to drop rare categorical data?
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 1, 2019 at 16:04 | history | edited | kjetil b halvorsen♦ | CC BY-SA 4.0 |
edited tags
|
| Jun 8, 2019 at 17:43 | comment | added | Peter Leopold | The threshold would be a meta parameter, so you would choose it by cross validation. That's the easy answer. In fact, that is how you would defend your choice. You would make your choice by looking at the exigencies of the rest of the calculation. Would threshold=1 be enough to stabilize the rest? Or 2? I would be quite greedy about the selection at first. Of course you would like it to be as small as possible, but first everything else has to work, so start higher rather than lower. 5? 10? | |
| Jun 8, 2019 at 17:00 | answer | added | Tim | timeline score: 2 | |
| Jun 8, 2019 at 16:17 | comment | added | Igor | By RareWord you mean some new binary feature that thresholds the counts of each categorical value? How would you recommend to choose this threshold if I understood you correctly? Thanks. | |
| Jun 8, 2019 at 16:07 | comment | added | Peter Leopold | I'd create a RareWord category. Infrequently used proper nouns -- like names -- are quite sensibly added to a RareWord category. In a posterior predictive exercise, a simulated sentence would give you something like "We asked Google to go on line and do a Bob search for a good restaurant." -- which is a perfectly good sentence without higher-order n-grams or context-sensitive assignments of rare words. That is, you may want to find digrams of rarer words that occur together, like "Google" and "search", in which case "Bob search" could be suppressed. | |
| Jun 8, 2019 at 15:33 | history | asked | Igor | CC BY-SA 4.0 |