Timeline for Feature selection on large file with missing categorical and numerical data in R
Current License: CC BY-SA 3.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 14, 2015 at 11:24 | comment | added | user90772 | Does it make sense to use the hashing trick to represent all the categorical multi-valued attributes with a binary number? | |
| Oct 14, 2015 at 8:58 | comment | added | user90772 | I found that I can use group lasso so that the algorithm will treat the whole group of binary encodings for the multi-valued attributes as a single predictor if I understand correctly. I am not sure if this is the way to go but I will give it a try. | |
| Oct 14, 2015 at 8:27 | comment | added | user90772 | Thank you both. My aim is to select a smaller set of features and then use a hierarchical model for predicting the distribution of the target variable by some mixture model (on a larger dataset). Another issue I have is that some of the predictors are multi-valued (i.e. comma-separated, variable number of the same set of values). For the moment I am encoding those using one-hot encoding but this also will not pick the attribute per se. Too many questions and I am a novice, apologies. | |
| Oct 13, 2015 at 21:26 | comment | added | EdM | Are you going to use the model for prediction, or for some other purpose? If for prediction, are there some variables that are most frequently missing so that they would not be available for prediction? | |
| Oct 13, 2015 at 18:35 | comment | added | Sergey Bushmanov | Keep it simple. Use stratified sampling, if you can, to make your sample representative for imputation techniques you apply. My point, again, your data set appears a little too big, at least for my laptop. On top of that, R is not very efficient in memory management. Make it sure your imputation algo works on a smaller sample, and then try increasing the size. | |
| Oct 13, 2015 at 18:06 | comment | added | user90772 | Thank you again, I have the same suspicion, but since I am a complete beginner, I guess I need to use some stratified sampling technique to get a representative sample, right? | |
| Oct 13, 2015 at 17:57 | comment | added | Sergey Bushmanov | If you suspect getting an error is a matter of imputation algo, you may try imputing on a fraction of data, say 1/10, but my suspicion you do not have enough RAM cause I estimate your data size is in Gb's | |
| Oct 13, 2015 at 17:32 | comment | added | user90772 | Thank you, I will update once I have the number. It takes a while to load the data. | |
| Oct 13, 2015 at 7:37 | answer | added | Chamberlain Mbah | timeline score: 1 | |
| Oct 12, 2015 at 18:22 | comment | added | Sergey Bushmanov | What is the RAM available prior to loading the data into R and RAM available after loading? | |
| Oct 12, 2015 at 16:55 | history | asked | user90772 | CC BY-SA 3.0 |