I am developing a text classification problem, in which at some time points, say at the end of each week, I receive a batch of feedback from users about correctly and wrongly classified inputs. I am trying different strategies for incorporating this feedback to improve the algorithm.
The base-line for me is the naive method of retraining a complete new model on the entire available data (including the new coming batch).
I have tried different strategies:
Assigning more weights to the miss-classified observations, and also more recent observations. In order to assign more weights I just copy the points with more weights to training data, e.g., if $x_1$ and $x_2$ have weights 1 and 2, respectively, the training data will be $(x_1, x_2, x_2)$.
Training a classifier on each coming batch and (with more weights on miss-classified observations) and then consider an ensemble of these simple classifiers, something like hard or soft voting for final decision.
Surprisingly the naive base-line eventually bits both methods. So the question is that why is this happening?
Here are the results. x-axis is the number of the coming batch, and the y-axis is the accuracy of the model on the new coming batch.
classification model is just a simple linear classifier as mean for comparison of methods for incorporating feedback.