I have 3 sets of data.
- A positively labeled dataset.
- An unlabeled dataset that has for sure positive (around 75%) and negative data.
- An unlabeled dataset that has for sure positive data and maybe negative data.
I'm interested in finding negative observations in this last dataset.
There will be no future new (test) data.
Im trying out some semisupervised logistic regression techniques using shift parameters (http://arxiv.org/abs/1305.4987).
Since I wont have future data should I include all my samples in building my classifier and classify my training data to determine the labels of dataset 2? Or should I train my classifier on only dataset 1 and 2 and then determine the labels of dataset 3?
If I include all data would I still have a problem with overfitting? Do I still need L1 regularisation or something?
- Is my choise of training algorithm appropiate for this problem?
- Im also strugeling what a good performance measure would be. Missclassification error is probably not ideal.