0
$\begingroup$

I know that there have been similar questions but most of them have not worked, this is why I start this question.

I have a very large dataset (around 2,500,000 records) with approximately 100 variables, most of which are categorical and some of them numerical. There are a lot of missing values which are probably MCAR/MAR (the data come from an automated log file). Also, some categorical variables have a lot of different values (e.g. city attribute). I am trying to perform feature selection to find the most informative variables with regard to a discrete numerical target variable.

I have two problems:

i) How to fill in the missing values: most of the standard packages' functions get stuck (i.e. mice, hmisc). Should I use just the mode/median or fill in some default "unknown" value? If I decide to take a subset of the rows to decrease the size, how small should this subset be - if there are any empirical rules?

ii) Most of the feature selection techniques I have tried (fscaret, fselector, boruta) after using some simple imputation technique either hang or give me an error. I believe this has to do with the fact that some categorical variables have many possible values. What are some practical ways to deal with this?

$\endgroup$
  • $\begingroup$ What is the RAM available prior to loading the data into R and RAM available after loading? $\endgroup$ – Sergey Bushmanov Oct 12 '15 at 18:22
  • $\begingroup$ Thank you, I will update once I have the number. It takes a while to load the data. $\endgroup$ – user90772 Oct 13 '15 at 17:32
  • $\begingroup$ If you suspect getting an error is a matter of imputation algo, you may try imputing on a fraction of data, say 1/10, but my suspicion you do not have enough RAM cause I estimate your data size is in Gb's $\endgroup$ – Sergey Bushmanov Oct 13 '15 at 17:57
  • $\begingroup$ Thank you again, I have the same suspicion, but since I am a complete beginner, I guess I need to use some stratified sampling technique to get a representative sample, right? $\endgroup$ – user90772 Oct 13 '15 at 18:06
  • $\begingroup$ Keep it simple. Use stratified sampling, if you can, to make your sample representative for imputation techniques you apply. My point, again, your data set appears a little too big, at least for my laptop. On top of that, R is not very efficient in memory management. Make it sure your imputation algo works on a smaller sample, and then try increasing the size. $\endgroup$ – Sergey Bushmanov Oct 13 '15 at 18:35
1
$\begingroup$

I had a similar problem. I used the $impute$ (from Hastie and Tibshirani) package that does K-Nearest Neighbor imputation. It was designed for micro-arrays, that means gene expression data which is continuous. But if you convert your predictors, all of them to numeric, it does a nice job and very fast.

source("bioconductor.org/biocLite.R") biocLite("impute")

$\endgroup$
  • $\begingroup$ Thank you Chamberlain, I will try this. I was wondering if you tried some feature selection on this dataset. $\endgroup$ – user90772 Oct 13 '15 at 8:45
  • $\begingroup$ Feature selection: the "LASSO" will be very suitable for this job. I am a big fan of Hastie and Tisbshirani. Use the $glmnet$ package. It will require patience. $\endgroup$ – Chamberlain Foncha Oct 13 '15 at 9:00
  • 1
    $\begingroup$ @ChamberlainFoncha Think twice before using Lasso for feature selection, it wasn't made for this. Lasso uses regularization to induce some bias to ensure better out-of-sample generalization of the Lasso model, not other models. If you carry over features selected by Lasso to other models, there is absolutely no guarantee this set of features will perform well. $\endgroup$ – Sergey Bushmanov Oct 13 '15 at 18:54
  • $\begingroup$ Feature selection with lasso is perfect. The parameter estimates are biased all in one direction. The selected set of predictors are the best. This subset is ordered that means if it is too small, reducing shrinkage will add more variables. The existing variables will remain in the set. You can then go ahead to do unbiased estimation with modeling without penalization on the subset if you are in for hypothesis testing. The lasso is perfect for feature selection. But if you need unbiased estimates, you already have the subset to work from. $\endgroup$ – Chamberlain Foncha Oct 13 '15 at 21:39
  • $\begingroup$ If you are in for prediction then you can use the lasso both for feature selection and parameter estimation. In that case your biased estimates will then have a small out-sample variance. Please note feature selection is a goal in itself. $\endgroup$ – Chamberlain Foncha Oct 13 '15 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.