For the past years I have been developing different type of Neural Networks with great success. For the past few weeks I have been working on a (big) project I am kind of struggling with. I hope by posting this question I might get some insights I did not think about yet. I feel like a lot is based on experience, and with this dataset I am lacking this specific experience.
I have the following dataset:
- 250K rows containing ~130 variables (90% continuous / 10% categorical)
- Each row only has about 10-15 of the variables (totally random) and is missing all the other variables
- We are predicting a binary response (0 or 1), where about 6% in the total dataset is positive (so classes are not evenly weighted)
So in summary, the 'pool' of variables is large and a lot of rows do not share the same variables with values. I know this results in a huge amount of missing values, but I hope the dataset size can kind of compensate this. In fact, missing a certain variable in a row might even have a relationship with the outcome. For now I feel like treating my missing data and choosing an appropriate design are crucial - but as I said I do not have a lot of experience with this kind of data. Any suggestions and insights from researchers with similar datasets would be highly appreciated. My main questions would be;
1. How should I clean/format my data, and more specifically what should I do with my missing values?
2. What type of neural network (or would any other ML algorithms such as random forest) and design would be most appropriate?
3. Any other suggestions/tips from your own experience?