For the past years I have been developing different type of Neural Networks with great success. For the past few weeks I have been working on a (big) project I am kind of struggling with. I hope by posting this question I might get some insights I did not think about yet. I feel like a lot is based on experience, and with this dataset I am lacking this specific experience.

I have the following dataset:

  • 250K rows containing ~130 variables (90% continuous / 10% categorical)
  • Each row only has about 10-15 of the variables (totally random) and is missing all the other variables
  • We are predicting a binary response (0 or 1), where about 6% in the total dataset is positive (so classes are not evenly weighted)

So in summary, the 'pool' of variables is large and a lot of rows do not share the same variables with values. I know this results in a huge amount of missing values, but I hope the dataset size can kind of compensate this. In fact, missing a certain variable in a row might even have a relationship with the outcome. For now I feel like treating my missing data and choosing an appropriate design are crucial - but as I said I do not have a lot of experience with this kind of data. Any suggestions and insights from researchers with similar datasets would be highly appreciated. My main questions would be;

1. How should I clean/format my data, and more specifically what should I do with my missing values?

2. What type of neural network (or would any other ML algorithms such as random forest) and design would be most appropriate?

3. Any other suggestions/tips from your own experience?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy