I was working on the scikit-learn-london dataset (on kaggle) and I was trying to do a binary classification task. The dataset is shortly described below:
- 40 features (all numerical, not categorical)
- 1000 training examples
- 9000 test examples
- labels 0 or 1
What I tried
I tried a bunch of classifiers and got approximately 90% cross validation accuracy. I used feature scaling, normalization, dimensionality reduction (with pca), feature selection and so on. The accuracy did not improve.
I also tried neural networks from 1 to 10 hidden layers, from very small to very large number of neurons. I used dropout and l2 regularizations. The accuracy did not improve.
If I'm not mistaken, theoretically, neural networks can fit every dataset with a good choice of parameters. So, I was about to conclude that this is the best accuracy one can get with this dataset.
What others did
Then I saw some people doing the following: Merge train and test datasets. Replace each row with gaussian mixture probabilities (without any extra preprocessing). These new features give around 99% accuracy with almost any classifier.
- How could these people foresee that replacing rows with gaussian mixture probabilities would work?
- Is this a standard step in data preprocessing, or somewhat revolutionary/brilliant idea?
- Merging train and test sets (only rows, not labels) seems a little bit of cheating to me, isn't it?
- Can I ever find a neural network architecture and hyperparameters that works just right for this problem? (without gaussian mixture)