1
$\begingroup$

My data is 1,785,000 records with 271 features. I'm trying to reduce number of features used to build the model.

Q1. while exploring the data I found that some features are almost all missing data, like only 25 records has value for this feature and the others records has missing values, so I thought that is not informative enough and it's better to eleminate those features, am I right? and if I am right, for what level I can do that, I mean if 90%, 80%, etc.. of each feature are missing values, when I can decide to get rid of these features? (taking in consideration that it is the dependent variable is N/Y and only %1.157 of the whole data is belonging to Y).

Q2. for each indivisual in the dataset, there are 64 trait_type listed, where each one can take one of the values [1 or 3 or 5]. my question is: if some trait-type take only value [5] or missing dat for all the record, does it have any value or again we can eliminate that feature?

Thank you

$\endgroup$
  • $\begingroup$ Have you had a look at some of the standard literature on this topic? For example: stat.columbia.edu/~gelman/arm/missing.pdf and other stuff cited in similar questions here. That might help you narrow your question down to something like 'should I use method x or y to deal with my missing data?' $\endgroup$ – Ben Apr 20 '12 at 6:08
1
$\begingroup$
  1. I don't think eliminating data is a good idea. Let me ask you this -- the features or variables that you are trying to eliminate, how do you know if they can be ignored or not? They could play vital roles in your model. So, I would consider imputation if I were you. The paper suggested by @Ben is good but this is also a great paper on missing data and multiple imputation. It will answer and/or guide you how to deal with imputing dependent variable as well.
  2. What are the values (1, 3 , and 5) mean? I have experience with imputation with only SAS PROC MI and if the observed values are all identical, like in your case (value=5), in any variables, PROC MI will automatically drop that variable and not impute it.
$\endgroup$
  • $\begingroup$ I thought of eliminating bcz the value is missing for ALL records except very few (like 20 records of 1,785,000 records). Also the validation data (9500 records) has totally missing data for that variable. It's loke I have this column with no information at all so I just can't understand how its presence will be valuable for training process! For Q2, [1 refers to music preference, 3 to reading preference, 5 to activity preference]. The goal is to predict if the user will response to email by click(Y) or notClick(N). No missing data for DV. Only nearly 1% in training set response by Y. $\endgroup$ – simplyme Apr 20 '12 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.