I am preparing a dataset for a binary classification. Some of the columns contain missing values.
I am thinking of either handling the missing values manually using pandas and replace each missing value by the mean of the column by class or using the
Imputer method by scikit.
Two questions, is it better to replace the missing values with the mean of the class rather than the mean of the whole column and if yes why isn't there such an option in scikit's
some data related info
For some features (sentiment analysis features) there a lot of missing values and that comes from the fact that some users (my training samples are pairs of users) don't share common hashtags (it's twitter a analysis task), but most of the features are topological so it doesn't make sense to discard all those lines.
On the other hand someone could say that since the users don't share any hashtags they should have a zero value for those features. But this isn't necessarily true. My crawling includes the latest 500 tweets for those users, not all tweets they have ever produced.
Also from visualizing the data I can see clear correlations between those features and the class values, from the values that are not gathered around zero due to lack of common hashtags.
Here are some visualizations
Here are two sentiment features, not exactly easily linearly separable but the problem can be observed (not exactly seen because of density), the stripe on the crossing of the axes holds close to half the values
Here is the same visualization but from weka this time