For the past years I have been developing different type of Neural Networks with great success. For the past few weeks I have been working on a (big) project I am kind of struggling with. I hope by posting this question I might get some insights I did not think about yet. I feel like a lot is based on experience, and with this dataset I am lacking this specific experience.

I have the following dataset:

  • 250K rows containing ~130 variables (90% continuous / 10% categorical)
  • Each row only has about 10-15 of the variables (totally random) and is missing all the other variables
  • We are predicting a binary response (0 or 1), where about 6% in the total dataset is positive (so classes are not evenly weighted)

So in summary, the 'pool' of variables is large and a lot of rows do not share the same variables with values. I know this results in a huge amount of missing values, but I hope the dataset size can kind of compensate this. In fact, missing a certain variable in a row might even have a relationship with the outcome. For now I feel like treating my missing data and choosing an appropriate design are crucial - but as I said I do not have a lot of experience with this kind of data. Any suggestions and insights from researchers with similar datasets would be highly appreciated. My main questions would be;

1. How should I clean/format my data, and more specifically what should I do with my missing values?

2. What type of neural network (or would any other ML algorithms such as random forest) and design would be most appropriate?

3. Any other suggestions/tips from your own experience?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.